Method and system for detecting and classifying objects in images, such as insects and other arthropods

ABSTRACT

A color-based imaging system and method for the detection and classification of insects and other arthropods are described, including devices for counting arthropods and providing taxonomic capabilities useful for pest-management. Some embodiments include an image sensor (for example, a digital color camera, scanner or a video camera) with optional illumination that communicates with a computer system. Some embodiments include a color scanner connected to a computer. Sampled arthropods are put on a scanner to be counted and identified. The computer captures images from the scanner, adjusts scanner settings, and processes the acquired images to detect and identify the arthropods. Other embodiments include a trapping device and a digital camera connected by cable or wireless communications to the computer. Some devices include a processor to do the detection and identification in the field, or the field system can send the images to a centralized host computer for detection and identification.

RELATED APPLICATIONS

Benefit is claimed under 35 U.S.C. 119(e) to provisional applicationSer. No. 60/478,636 entitled “DEVICES, SOFTWARE, METHODS AND SYSTEMS FORELECTRONIC OBJECT DETECTION AND IDENTIFICATION AND APPLICATION TO THEDETECTION OF INSECTS AND OTHER ARTHROPODS” by Val R. Landwehr andFernando Agudelo-Silva, filed Jun. 13, 2003, which is incorporated inits entirety by reference.

FIELD OF THE INVENTION

The present invention relates to the field of automated machine-visionrecognition, and more specifically, to a method and apparatus formachine-vision object detection and classification, particularly ofinsects and other arthropods.

BACKGROUND

Timely, practical and accurate detection and classification ofarthropods is crucial in many instances. There are many species ofarthropods, particularly among the insects and mites, that causesignificant damage and loss to plants, wood and fiber and transmitpathogens among people and other animals. The efficient, accurate andtimely detection of arthropod pests is a key factor in managing theirpopulations and limiting the damage and injury they cause. Detection isnecessary to determine: 1) arthropod presence or absence; 2) theirclassification to a certain taxonomic category such as genus or species;3) their relative or absolute numbers; 4) a critical period in thearthropod pest's life cycle that is amenable to control measures; and,5) significant phases in the relationship between the arthropod and theorganism that it affects.

Estimates of arthropod pest numbers are necessary to decide whethercontrol measures are warranted and detection of the various life stagesof a pest suggests when control techniques will be most effective.Associating pest numbers and the pest's life cycle to periods when thehost is most vulnerable to injury is also critical in pest management.In addition to insect pests there are many beneficial insect, spider andmite predators that need to be sampled as part of a pest managementprogram. There is also need for a more expeditious technology toclassify arthropods in ecological studies. Thus, the sampling ofarthropod populations in various habitats is an integral part of suchdiverse fields as ecological studies, crop protection and human health.

SUMMARY OF INVENTION

Several embodiments of machine-implemented, image-based systems fordetecting and classifying insects and other arthropods are described.Examples of useful and practical applications of the systems aredescribed. These examples show that the present invention provideslabor-saving devices for counting arthropods and provides improvedtaxonomic capabilities for pest management specialists, ecologists,science educators, agricultural extension and inspection agencies, amongothers.

In some embodiments, a sticky substrate is provided in order thatarthropods to be classified are captured. In some embodiments, thesticky substrate has a first area that has a first background color (forexample, white or bright yellow) and a second area that has a secondcontrasting background color (for example, black or dark blue). Such asubstrate having a plurality of different colors is useful for obtainingimages of arthropods having different colors. For example, small whitethrips are difficult to detect on a white background or even on a yellowbackground, however on a black or dark blue background they are mucheasier to detect. Some embodiments use various graphical patterns,specific color(s), pheromones, kairomones, and/or other chemicalattractants to lure the arthropods to the collection surface. In someother embodiments, arthropods are collected and either killed orimmobilized and then they are placed on a detection surface which neednot be sticky.

A digital camera, flat-bed scanner or other suitable imaging device isused to capture an image of the substrate along with any arthropods thatmay be stuck to it. In some embodiments, the image is obtained in thefield (at the point of collection); in other embodiments, the stickycollection surface with its attached arthropods is transported, mailed,or taken to a facility where the imaging takes place. In someembodiments, an initial reference image of the substrate background isobtained, then insects or other arthropods are collected and anotherimage is obtained, in order to use the difference between the two imagesto calibrate colors and/or to more readily detect the newly capturedarthropods as difference areas between the two images. In someembodiments, a plurality of images of the same substrate is obtainedover time, wherein the incremental differences in the images providesinformation as to when each arthropod appeared.

Once the image or images are obtained, each image is analyzed to detectpixels of interest, to group the detected pixels into detected objects,and the detected objects are processed to extract image information,such as a hue and saturation histogram, the length, width, length-widthratio, perimeter measurement, and/or certain patterns or locations ofcolor information within the detected object, and this image informationis compared to a set of reference image information collected frompre-identified arthropods in order to determine which, if any, of thereference arthropods most closely matches the object to be identified.

In some embodiments, once the identification or classification has beenmade, this information is entered into a database (a collection ofordered information), that tracks such information as the date andlocation of collection, which and how many of each type of arthropod wascollected. In some embodiments, the database also collects andcorrelates other information such as the types of crops or othervegetation in the area of collection, the types of insecticides used andwhen, and other information that could be useful in arthropod managementprograms. In some embodiments, from this information, reports aregenerated and communicated to relevant governmental (e.g., county,state, or federal) or commercial entities (e.g., grower's associations,coops, or pest-management consultants).

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 is a flowchart of a method 100 according to some embodiments ofthe invention.

FIG. 2A is representation of a data structure 200 used in someembodiments of the invention.

FIG. 2B is representation of a data structure 250 used in someembodiments of the invention.

FIG. 3 is a perspective block diagram 300 of a system used to acquire animage in some embodiments of the invention.

FIG. 4A is a representation of a detected-object-pixels data structure400 used in some embodiments of the invention.

FIG. 4B is a representation of a silhouette-pixels data structure 401used in some embodiments of the invention.

FIG. 4C is a representation of an outline silhouette-pixels datastructure 402 used in some embodiments of the invention.

FIG. 4D is a representation of an outline silhouette-pixels datastructure 412 used in some embodiments of the invention.

FIG. 4E is a representation of a silhouette-pixels data structure 422used in some embodiments of the invention.

FIG. 4F is a representation of a silhouette-pixels data structure 432used in some embodiments of the invention.

FIG. 4G is a representation of a reference silhouette-pixels datastructure 440 used in some embodiments of the invention.

FIG. 5 is a flowchart of a method 500 according to some embodiments ofthe invention.

FIG. 6 is a perspective block diagram 600 of a system used to acquire,detect, and classify arthropods, in some embodiments of the invention.

FIG. 7 is a list of a method 700 according to some embodiments of theinvention.

FIG. 8A is a flowchart of a method 800 according to some embodiments ofthe invention.

FIG. 8B is a flowchart of a method 810 according to some embodiments ofthe invention.

FIG. 8C is a flowchart of a method 804 according to some embodiments ofthe invention.

FIG. 8D is a flowchart of a method 805 according to some embodiments ofthe invention.

FIG. 8E is a flowchart of a method 806 according to some embodiments ofthe invention.

FIG. 8F is a flowchart of a method 807 according to some embodiments ofthe invention.

FIG. 9 is a perspective block diagram of a system 900 used to acquire animage in some embodiments of the invention.

FIG. 10A is a representation of a calibration surface 915 used in someembodiments of the invention.

FIG. 10B is a graph of an example calibration function 1010 used in someembodiments of the invention.

FIG. 10C is a block diagram of a collecting chamber 1020 adapted to orcoupled with vacuum device(s) to sample insects.

FIG. 10D is a perspective view of a sample cleaning system 1030 used insome embodiments.

FIG. 10E is a perspective view of a sample-processing unit 1040 used insome embodiments.

FIG. 10F is a perspective view of a set of scanner lids 1050 used insome embodiments.

FIG. 10G shows a block diagram of an example on-linearthropod-identification service 1070.

FIG. 10H shows an example reference database structure 1060 for keyarthropods.

FIG. 10I shows a first portion of an example referencestatistical-feature database structure 1080 for key arthropods.

FIG. 10J shows a second portion of the example referencestatistical-feature database structure 1080.

FIG. 10K shows a first portion of an example referencestatistical-feature database definition 1081 for key arthropods.

FIG. 10L shows a second portion of the example referencestatistical-feature database definition 1081.

FIG. 10M shows a first portion of an example reference color-silhouettedatabase definition 1082 for key arthropods.

FIG. 10N shows a second portion of the example referencecolor-silhouette database definition 1082.

FIG. 11 is a flowchart of a method 1100 according to some embodiments ofthe invention.

FIG. 12 is a flowchart of a method 1200 according to some embodiments ofthe invention.

FIG. 13 is a flowchart of a method 1250 according to some embodiments ofthe invention.

FIG. 14 is a flowchart of a method 1260 according to some embodiments ofthe invention.

FIG. 15 is a flowchart of a method 1500 according to some embodiments ofthe invention.

FIG. 16 is a flowchart of a method 1600 according to some embodiments ofthe invention.

FIG. 17 is a flowchart of a method 1700 according to some embodiments ofthe invention.

FIG. 18 is a flowchart of a method 1800 according to some embodiments ofthe invention.

FIG. 19 is a flowchart of a method 1900 according to some embodiments ofthe invention.

FIG. 20 is a flowchart of a method 2000 according to some embodiments ofthe invention.

FIG. 21 shows a portion of YCbCr space.

FIG. 22 shows 2D hue/color saturation histogram for a halictid bee.

FIG. 23 shows values of the circular fit/compactness feature for threeclasses of geometric shapes.

FIG. 24 shows a flowchart 2400 of the general description of theoperation of the system.

FIG. 25 shows a digital image called ScanDorsalTraining.bmp used forgenerating the identifying reference features.

FIG. 26 shows a digital image ScanVentralTraining.bmp that has theventral view of the same eleven individuals as FIG. 25.

FIG. 27A is a test image ScanDorsalTest.bmp of ten insect individuals.

FIG. 27B is a test image ScanVentralTest.bmp of the same ten insectindividuals as FIG. 27A.

FIG. 28A is an image having dorsal views of the same ten garden insectsas FIG. 27A.

FIG. 28B is the image after successful detection and recognition ofthese insects.

FIG. 29A is an image having ventral views of the same ten garden insectsas FIG. 27B.

FIG. 29B is the image after the successful detection and recognition ofthese insects.

FIG. 30A is a test image of insects in clutter.

FIG. 30B is the output results image with the correct detection andidentification of the objects.

FIG. 31 is an image that simulates a snapshot from a previous samplingperiod.

FIG. 32A is an image of the syrphid fly species with a striped thorax.

FIG. 32B is an image of an asparagus beetle.

FIG. 32C is an image of a second species of syrphid fly with no stripeson the thorax.

FIG. 32D is an image of a halictid bee.

FIG. 32E is an image of a blow fly.

FIG. 32F is an image of a multicolored Asiatic ladybird beetle.

FIG. 33 is a test image of insects being overlapped by other insects orclutter.

FIG. 34 is an image after successful detection and classification in thecase of occlusion.

FIG. 35A is the silhouette of the occluded bee.

FIG. 35B shows the silhouette of the halictid bee prototype.

FIG. 35C shows an occluded halictid bee's silhouette matched best with aprototype silhouette of a halictid bee.

FIG. 36 is an image showing color coding of best matches for three casesof occlusion.

FIG. 37A shows silhouette matching results for occlusion of twoasparagus beetles.

FIG. 37B shows silhouette matching results for occlusion of twoasparagus beetles.

FIG. 37C shows silhouette matching results for occlusion of two ladybirdbeetles.

FIG. 37D shows silhouette matching results for occlusion of two ladybirdbeetles.

FIG. 37E shows prototype silhouette and spotprints for a halictid bee.

FIG. 37F shows silhouette matching results for a halictid bee occludedby an ash seed.

FIG. 38A shows a spurious correlation of the silhouette matches for theoccluded bee.

FIG. 38B shows the correct correlation of the silhouette matches for theoccluded bee.

FIG. 38C shows a spurious correlation of the silhouette matches for theoccluded bee.

FIG. 39 shows equipment setup used in some embodiments for testing.

FIG. 40 shows a detection surface before weevils are “collected” orplaced.

FIG. 41 shows an image of seven boll weevils used for training theclassifier.

FIG. 42 shows an image of detection surface after simulated collectionof three weevils.

FIG. 43 shows an image of detection surface after “collecting” threeadditional insects.

FIG. 44 shows an image output following processing.

FIG. 45 shows an image output following processing.

FIG. 46 is a graph of histograms showing distribution of the referenceboll weevils (training weevils), unknown or test weevils and thecantharid beetle.

In the drawings, like numerals describe substantially similar componentsthroughout the several views of the process of being made. Signals andconnections may be referred to by the same reference number, and themeaning will be clear from the context of the description.

DETAILED DESCRIPTION

In the following detailed description of preferred embodiments,reference is made to the accompanying drawings that form a part hereof,and in which are shown, by way of illustration, specific embodiments inwhich the invention may be practiced. It is to be understood that otherembodiments may be utilized and structural changes may be made withoutdeparting from the scope of the present invention.

Definition: Object “identification” includes the detection andclassification (such as the name or other identification information,type, species and genus, age, developmental stage) of an object, such asof an arthropod.

FIG. 1 is a flowchart of a method 100 according to some embodiments ofthe invention. Method 100 includes operation 110 of acquiring an image,operation 120 of detecting an object in the image, operation 122 ofmatching color information from the object to a database of arthropodcolor information, operation 124 of outlining the silhouette of theobject, operation 126 of mapping the object outline to a standardorientation, operation 130 of matching the outline geometry of theobject to reference outlines, operation 140 of matching the colorgeometry from the object to references, and operation 150 of enteringthe classification and/or count into a database of detected arthropodsoptionally with other information (e.g., location, date, and timeinformation).

Note that operations 126 and 130, in some embodiments, map theorientation of the unknown object to a standard orientation (e.g.,rotating and/or translating (sliding side-to-side and up or down) untilthe longest dimension and an origin are aligned to an X-Y coordinatesystem) and then compare that outline to each reference outline from areference library (i.e., database) of outline geometries of arthropodseach of which is in that standard orientation until a match is found. Inother embodiments, for each reference outline geometry from thereference library, the reference outline is obtained one at a time, andthe reference outline is rotated and/or translated, compared to theoutline of the unknown object (which is in the random orientation inwhich it was detected), then the reference outline is rotated/translatedmore, again compared, and so on until the best match is found. Thus,either the unknown can be rotated/translated, and then compared to thelibrary of reference outlines; or the reference outlines can berotated/translated and compared to the outline of unknown.

At block 110, some embodiments of method 100 include acquiring an image,for example, obtaining a digital image from a scanner or digital camerathat “looks” at a sticky substrate, possibly having one or morearthropods that are to be classified, (i.e., “identified” or“recognized”). At block 120, some embodiments of method 100 includedetecting an object, i.e., distinguishing pixels of the object from abackground, and then grouping or associating neighboring pixels as asingle detected object, or “detection.” At block 122, some embodimentsof method 100 include matching the color histogram (e.g., how manypixels, regardless of location within the image, are of a particular hueand saturation) from the object to histogram data of a referencearthropod from a database having extracted information from a pluralityof pre-identified arthropods.

Some embodiments match primarily on the basis of the matching done atblock 122 and other feature matchings, and omit the matching operationsof blocks 126, 130, and/or 140. At block 124, some embodiments of method100 include outlining the detected object, i.e., determining whichpixels form the outer boundary or silhouette of the detection. At block126, some embodiments of method 100 include mapping (i.e., rotating andtranslating pixels) the arthropod's outline to a standard orientation(e.g., head up and image centered). At block 130, some embodiments ofmethod 100 include matching the outline geometry (silhouette) from theobject to a particular reference silhouette. At block 140, someembodiments of method 100 include matching the color geometry from theobject (e.g., whether particular pixels, at a particular location (e.g.,X and Y offsets from an origin location) within the image, are of aparticular hue and saturation). At block 150, some embodiments of method100 include entering the classification and/or the count of arthropodsof a particular classification into a database of detected arthropodsoptionally including the location, date, time, environment, or othercollection data. In other embodiments, the classification is output to auser.

FIG. 2A is a representation of a data structure 200 (e.g., useful in adatabase, in some embodiments). In some embodiments, data structure 200includes reference database information for key arthropods. Someembodiments include one or more different reference databases ofimportant arthropods for classification. In some embodiments, aplurality of similarly structured databases are provided. Each suchdatabase is tailored for particular agricultural crops (e.g., for fielduse) and/or commodities (e.g., for use in grain elevators or othercommodity-storage facilities), or other specialized or identifiedenvironments. Each database structure 200 contains a plurality ofrecords 221, 222, etc., that include a sufficient representation of thevariation in appearance of the important and common arthropods for aparticular crop or environment. In some embodiments, an entry in thedatabase includes an identification number 201, color information 232,color geometry information 236, outline geometry information 241, etc.In some embodiments, the color information 232 includes luminanceinformation 233, hue information 234, and saturation information 235. Insome embodiments, a plurality of subfields 231 (sometimes called a“spotprint”) are provided, each having color information 232 (e.g., hueand saturation) and color-geometry information 236 (e.g., the X and Yoffset for that hue and saturation). Some embodiments include furtheridentification data 242 (such as a complete reference image to beprovided to the user to assist the user in visually confirming aclassification from the system).

FIG. 2B is a representation of a data structure 250 (e.g., useful in adatabase, in some embodiments). In some embodiments, data structure 250includes a plurality of entries 261, 262, etc., each with informationfor each different type or group of arthropods that have been identified(e.g., entry 261 for the first type of arthropod identified, and entry262 for the second type of arthropod identified). In some embodiments,the entries in the database include taxonomic information such asgenus-species identification information 251, age and/or developmentalstage information 259, location found information 252, date detectedinformation 253, time detected information 254, lure information 255,count information 265, etc. Some embodiments of data structure 250 areimplemented as a relational database. Some embodiments of data structure250 are implemented as a relatively simple table. Some embodiments ofdata structure 250 include further reference information that can beprovided to a user such as images of each species and/or eachdevelopment stage so that the human user can do a visual double check ofthe results from the automatic system, or control methods, or otherinformation useful to the user and correlated to the identificationsmade.

FIG. 3 is a high-level perspective block diagram 300 of a system used toacquire an image in some embodiments of the invention. The systemincludes a surface 89 used to trap and/or attract arthropods 98. In someembodiments the arthropods are manually collected, killed or immobilizedand then placed on a detection surface in order to be imaged. Imagersystem 310 captures a digital image of the surface 89 and arthropods 98,and transfers the image by cable or wireless communications todata-processing system or computer 320 for detection and classification.Some embodiments of the invention include (see FIG. 3) acomputer-readable media 321 (such as a diskette, a CDROM, a DVDROM,and/or a download connection to the internet) having instructions storedthereon for causing a suitably programmed data processor to execute oneor more of the methods described in FIG. 1 and below, and/or havingdatabase records stored thereon such as described for FIGS. 2A, 2B, 10H,10I, 10J, 10K, 10L, 10M, 10N and elsewhere herein.

FIG. 4A is a representation of detected-object-pixels data structure 400used in some embodiments of the invention. In some embodiments, digitalimage 400 is in gray-scale while in other embodiments, includes colorinformation. Data structure 400 represents an example of the digitalimage captured by a camera of an arthropod to be classified.

FIG. 4B is a representation of a filled-silhouette-pixels data structure401 used in some embodiments of the invention. The captured digitalimage has been processed to isolate subject arthropod from thebackground image, and is represented in filled-silhouette form 401(e.g., wherein all the background pixels are set to a zero value (black)and all the data pixels are set to a 255 value (white)).

FIG. 4C is a representation of an outline-silhouette-pixels datastructure 402 (or simply called a silhouette data structure) used insome embodiments of the invention. The digital image 401 has beenfurther processed to convert it to an outline silhouette 402 with acenter-of-mass of point 410.

FIG. 4D is a representation of a slightly rotated silhouette-pixels datastructure 412 used in some embodiments of the invention. The digitalimage 402 has been rotated around the center-of-mass 401 (i.e., thegraphic center of the silhouette) to produce digital image 412 with acenter-of-mass 411.

FIG. 4E is a representation of a rotated silhouette-pixels datastructure 422 used in some embodiments of the invention. The digitalimage 402 has been rotated further around the center-of-mass 401 toproduce digital image 422 with a center-of-mass 421.

FIG. 4F is a representation of a further-rotated silhouette-pixels datastructure 432 used in some embodiments of the invention. The digitalimage 402 has been rotated still further around the center-of-mass 401to produce digital image 432 with a center-of-mass 431. The rotation ofthe image 402 around the center-of-mass 410 continues until the image isin a standard orientation as shown in image 432.

FIG. 4G is a representation of a reference silhouette-pixels datastructure 440 used in some embodiments of the invention. Reference datastructure 440 consists of the outline silhouette 442 and thecenter-of-mass 441. In some embodiments, the unknown image datastructure in standard orientation, such as in image 432, is comparedagainst the reference data structure 440 to determine a best match andto classify the unknown arthropod. In some other embodiments theoutlines of reference data structures are initially in a standardorientation but they are rotated and translated to compare with theoutline of the unknown data structure that, because of the random natureof collection, may not be in a standard orientation.

FIG. 5 is a high-level flowchart of a method 500 according to someembodiments of the invention. In some embodiments, method 500 representsa high-level overview of method 100 of FIG. 1, and of the methods ofFIGS. 10-20 and 24. Method 500 includes the process of acquiring theimage 510 (input), processing the image to detect and identify thearthropods 520 (process), and outputting or transmitting data thatidentifies and quantifies the detected and identified arthropods 530(output).

FIG. 6 is a perspective block diagram 600 of a system used to acquire,detect, and classify arthropods, in some embodiments of the invention.In some embodiments, system 600 includes an image acquisition system 310and an image processing system 320, as depicted in FIG. 3.

In some embodiments, e.g., particularly for deployment in the field,trapping-and-image-acquisition system 610 includes a color digitalcamera 611 having a lens 612 is connected by cable, fiber, or wirelesscommunications 631 (such as over the internet) to thecommunications-receiving hardware (e.g., an input/output subsystem) 630of the user's host computer 320. The camera 611 takes images ofarthropods (e.g., 91, 92, 93, and 94) on a trapping surface 624 which ispart of the device. In some embodiments, thetrapping-and-image-acquisition system 610 includes a filter 614 over thelens 612 and one or more illumination devices 613 to enhance the imagesof the arthropods of interest. Some embodiments include a diffuser orsimilar device (not explicitly shown, but for example, by havingdiffuse-type LEDs for lights 613) on the illumination devices 613, inorder to reduce shadows and make the illumination more even across theentire surface 624. In some embodiments, the trapping andimage-acquisition system 610 can include a pre-processor to do thedetection and classification in the field. In other embodiments, thesystem 600 sends the images to the user's host computer for detectionand classification. In some embodiments, the user initiates a directrequest from the host computer 320 for an image or to schedule periodicimage sampling and uploading.

In some embodiments, system 600 is used for laboratory and other indoorapplications. In some embodiments, image-acquisition system 620 includesa scanner 621. On the surface of the scanner, in some embodiments, a box622 is used to elevate the sampling substrate or background 624 slightlyabove the scanner surface (e.g., so the sticky paper does not stick tothe glass scanning surface or to filter 623, if used). A filter 623(e.g., color and/or polarization filter) can optionally by used toenhance the image of the arthropods of interest. In some embodiments,the user places or attracts sampled arthropods onto a substrate 624 tobe entrapped and/or scanned, in order to have them detected, counted,and classified. In some embodiments, substrate 624 is sticky, in orderto entrap the arthropods. In some embodiments, substrate 624 is coloredor patterned with a background that has been empirically determined toattract the arthropods of interest. In some embodiments, substrate 624is colored or patterned with a plurality of different colors in order tohave different contrasting backgrounds that enhance the contrast betweenthe background and a plurality if different arthropods of interest. Insome embodiments, substrate 624 includes a chemical attractant to lurethe arthropods to its surface. The scanned image of the background andany arthropods that may be on the substrate is sent/transmitted by cable632, or wireless communications, or other suitable means (such asmailing a CDROM having the image data stored thereon) to thecommunications hardware 630 and host computer 320.

In some embodiments, host computer 320 contains software to capture orreceive images from the camera system 610 and/or the scanner system 620,and process the acquired images to detect and classify the arthropods.The host computer 320 software processes the images 640 and producesoutput identification data 660 and/or updates database records witharthropod information 650, including, in some embodiments, entering datainto database 651, including location data 652 (state, county, field,location within field), date 653, conditions 654, insectidentifications, counts, etc. 655.

In some embodiments, the image processing 640 includes locating theobjects in the camera or scanned digital image 641, isolating theobjects in the image 642, reorienting the object in a standardorientation 643, comparing the object to a reference object or adatabase of reference objects 644, and identifying the object bydetecting and classifying it. The results of the image processing 640optionally result in output identification data 660 and/or updatesdatabase records with arthropod information 650. In some embodiments,updating the database records includes entering the data in a database651. The data in the database can include location data 652 (e.g.,state, county, farm name, field location, GPS location, etc.), date ofsampling or processing 653, conditions of sampling 654, along with thenumber of detected arthropods and the arthropod identification andclassification information 655. In some embodiments, the right-hand half(data processing portion) of FIG. 6 is equivalent to system 320 of FIG.3, and the left-hand half (image obtaining portion) of FIG. 6 isequivalent to system 310 of FIG. 3.

FIG. 7 is a flowchart/list of a method 700 according to some embodimentsof the invention. Method 700 includes the process of obtaining an image710, isolating the object image from the background image using “color”720 (e.g., luminance and/or hue and saturation), isolating the objectsin the image 730 (e.g., object A from object B), generating imageattributes 740 of object A, comparing image attributes to database ofreference attributes 750, storing output detection and identificationinformation for each arthropod 760 (e.g., nearest matches and/orconfidence levels), entering the arthropod classifications 770.

FIG. 8A is an overview flowchart of a method 800 according to someembodiments of the invention. At blocks 99 and 98 respectively, someembodiments of method 800 include inputting or obtaining the currentimage (the one to be analyzed), and inputting or obtaining the prior orbackground image (the one to be subtracted from the current image). Atblock 810 (further described in FIG. 8B and FIG. 8C), some embodimentsof method 800 include enhancing the image (e.g., by correcting colorsbased on an subset of the image that represents a calibration standard).At block 830 (further described in FIGS. 8C and 8F), some embodiments ofmethod 800 include segmenting of the detected objects (i.e., collectingor associating pixels that appear to be of a single detection (ordetected object)). Blocks 810 and 830 represent the data-independent orpixel-level processing 801. At block 840 (further described below inFIG. 8D and FIG. 8E), some embodiments of method 800 include extractingfeatures (e.g., finding the color histogram and silhouette, and/orrotating/translating to a standard orientation). At block 850 (furtherdescribed below in FIG. 8D), some embodiments of method 800 includestatistically classifying the objects (e.g., arthropods or other objectsof interest) using their extracted features. At block 860 (see FIG. 8D),some embodiments of method 800 include syntactically classifying usingsilhouette and/or color-reference-pixel matching. At block 870, someembodiments of method 800 include updating an arthropod database and/oroutputting the classification obtained. Blocks 840, 850, 860 and 870represent the data-dependent or symbolic-level processing 802.

FIG. 8B is a flowchart of a method 810 according to some embodiments ofthe invention. At block 811, some embodiments of method 810 includeenhancing the image using noise reduction (e.g., temporal averagingand/or spatial averaging), and/or perspective distortion correction(e.g., mapping pixels to a normalized view) and/or color correction(e.g., adjusting the gamma or contrast to obtain more correct colorrenditions). At block 814, some embodiments of method 810 include block815 of transforming data from an RBG format to a YCbCr color space,which includes block 816 of calculating the intensity image (the Ydata), block 817 of calculating the hue image (the arctangent of (Cr/Cb)data), and block 818 of calculating the saturation image (the squareroot of (Cr squared plus Cb squared) data).

FIG. 8C is a flowchart of a method 804 of low-level (or pixel-level)image processing, according to some embodiments of the invention.Low-level processing means that each function is applied to each pixel.At block 90, some embodiments of method 804 include acquiring the image.The acquired image can be either a color or black-and-white (B&W) image.For B&W images, the processing skips functions involving hue andsaturation images as well as the calculation of color features. At block811, some embodiments of method 804 include optional enhancing of theimage as described in FIG. 8B above. At block 813, some embodiments ofmethod 804 include calculating a background image from the current image(e.g., determining what color most of the pixels are in a given area,and using that color as the background for that area). At block 812,some embodiments of method 804 include choosing which background imagetype to use. At block 814, some embodiments of method 804 includecreating intensity, hue, and saturation images for the current andbackground images, as described in FIG. 8B above. At block 819, someembodiments of method 804 include creating difference images (betweenthe current and background images) for each of the three image types(intensity, hue, and saturation), and producing outputs 820. At block821, some embodiments of method 804 include performing an adaptivesearch for a segmentation threshold for each of the three image types.At block 822, some embodiments of method 804 include applying thresholdsto produce three types of segmentation images. Block 830 includes blocks831 and 832. At block 831, some embodiments of method 804 includeapplying combined segmentation logic and pixel-level shadow rejection toproduce a segmented image. At block 832, some embodiments of method 804include labeling regions (using connected components logic) and/orrejecting small clutter, to produce a labeled image 839. The labeledimage generated by region labeling is also called connected components.Control then passes to FIG. 8D.

FIG. 8D is a flowchart of a method 805 of high-level (or object-level)image processing, according to some embodiments of the invention.High-level processing means that each function is applied to each objector detection. At block 840, some embodiments of method 805 includeextracting features and/or silhouettes. At block 850, some embodimentsof method 805 include performing statistical classification. At branchblock 857, some embodiments of method 805 include going to block 871 ifthe best match is “good” and by far the best; else control passes toblock 858. At block 871, some embodiments of method 805 includeincrementing the appropriate species counter and/or updating graphicaloutput. At branch block 858, some embodiments of method 805 includegoing to block 872 if the method is not executing the option ofsilhouette matching; else control passes to block 860. At block 872,some embodiments of method 805 include incrementing the classificationcounter for “other” (for detected objects that were not matched to anyreference item included in the reference database) and/or updating thegraphical output. At block 860, some embodiments of method 805 includeperforming silhouette matching to discern closely ranked species, orpossible occlusions, or incomplete or damaged arthropod bodies. Atbranch block 867, some embodiments of method 805 include going to block873 if the best silhouette match is “good”; else control passes to block868. At block 873, some embodiments of method 805 include incrementingthe appropriate species counter and/or updating graphical output. Atblock 868, some embodiments of method 805 include rejecting the detectedobject as clutter or “other” and going to block 874. At block 874, someembodiments of method 805 include incrementing the classificationcounter for “other” and/or updating the graphical output.

FIG. 8E is a flowchart of a method 806 of an arthropod classificationprocess used as an alternative or supplement to that of FIG. 8Daccording to some embodiments of the invention. At block 840, someembodiments of method 806 include extracting statistical features (e.g.,size, shape, perimeter length, intensity, color—e.g., histograminformation) and or extracting silhouette data (e.g., perimeter pixels(the outline), and/or color-reference pixels). At block 851, someembodiments of method 806 include performing a 1NN classification usinga feature-reference database 91. At branch block 856, if the 1NNdecision is “good,” control passes to block 875 where the classificationis output; else control passes to block 861. At block 861, someembodiments of method 806 include silhouette matching using referencedata from prototype silhouette database 92. At block 876, someembodiments of method 806 include outputting the resultingclassification.

FIG. 8F is a flowchart of a method 807 that performs segmentationprocessing according to some embodiments of the invention. In someembodiments, data 820 is obtained from block 819 of FIG. 8C. At block823, some embodiments of method 807 include pixel labeling using anadaptive threshold from a histogram search. Block 830 includes blocks831, 834, and 835. At block 831, some embodiments of method 807 includeapplying segmentation logic using a modified OR of the detected pixels.At block 834, some embodiments of method 807 include performingmorphological operations, such as filling in holes within detectedobjects or smoothing the edges of detected objects. At block 835 (as in832 of FIG. 8C), some embodiments of method 807 include performingconnected-components logic to obtain a labeled image.

FIG. 9 is a perspective block diagram of a system 900 used to acquire animage in some embodiments of the invention. System 900 includes, in someembodiments, a collection surface or substrate 910 having a stickysurface over at least part of its surface area, in order to capturearthropods, and having a plurality of different backgrounds (e.g.,having different colors, hues, shades, and/or patterns) that enhanceimage quality and contrast for a variety of different arthropods, helpcalibrate imager color or contrast, and/or attract or repel variousarthropods. For example, some embodiments include a white area 911(useful for good image contrast with some black or darker arthropods), ayellow area 913 (useful for attracting certain arthropods), a blue area912 and/or a black area 914 (useful for good contrast with some white orlighter arthropods), and/or an area 919 having a striped, spotted,checkered, wavy or other pattern(s) that has been empirically determinedto attract (in order to capture certain varieties that the user desiresto observe) or repel certain varieties of arthropods (in order to avoidcapturing other varieties that the user desires not to have in her orhis images). It may be that a certain color (e.g., a particular shade ofgreen) is useful to attract the prey to the trap, but that perhaps adifferent color is a better background for obtaining images, and thus,in some embodiments, both colors are provided on the collection surfaceor within the trap. Some embodiments also include a scale 909 that isuseful to adjust the size of the image to a standard metric.

Some embodiments include a calibration patch 915 (which, in someembodiments, is not sticky in order to avoid having arthropods or debrisblocking portions of its image), wherein patch 915 includes a pluralityof different colors, hues, or shades 916, 917, and/or 918, useful tocalibrate the image obtained for later preprocessing to obtain moreaccurate color renditions. Some embodiments include side lighting 920(provided, e.g., by one or more LEDs) and/or front lighting 921 (alsoprovided, e.g., by one or more LEDs) that are used either together toobtain a well lit image without shadows, or separately (e.g.,alternately) to obtain one or more images having differing lightingconditions to obtain images one of which might have better image qualitythan others. In other embodiments, available sunlight is used instead.Some embodiments include one or more diffusers (not shown) on, or infront of, the LEDs in order to further reduce shadows. Some embodimentsinclude a colored filter 931 (e.g., a red or pink filter in someembodiments, to reduce contrast of those colors and/or increase thecontrast of complementary colors) and/or a polarizing filter 932 (e.g.,to reduce glare) (note that the horizontal-line pattern on filter 932does not necessarily represent the color blue (such as in patch 912),but rather an exemplary polarization direction). Some embodimentsinclude an imager 310. Some embodiments include an enclosure 960 (shownin dotted lines) to hold or support the other components of system 900.Some embodiments include a substrate or container 970 having a chemicalattractant (such as a pheromone and/or kairomone) to attract a widevariety of arthropods, or to selectively attract only certain types,and/or having a chemical repellant to selectively avoid capturingcertain types of arthropods. In some embodiments, the chemicalattractant substrate 970 is included as a portion (i.e., unified with)background substrate 910. In other embodiments, a separate container isprovided as shown in FIG. 9. In some embodiments, substrate or container970 is made onto and sold as part of substrate 910. In otherembodiments, substrate or container 970 is separately sold and thenplaced on or near substrate 910 in the field. In some embodiments,substrate 910 is a consumable item that is purchased separately andperiodically replaced. In some embodiments, substrate 910 andcalibration patch 915 are sold or delivered separately, and then eitherused separately within the imaging field-of-view, or stuck together asshown.

Some embodiments include standardized consumable sticky sheets 910 fortrap system 900. These provide sticky coated sheets for trappinginsects. In some embodiments, cards come in several sizes to accommodatestandard pheromone traps, customized traps and simple sticky boards. Insome embodiments, the sticky material is impregnated with variousattractants such as pheromones, kairomones, plant or microbial extracts.In some embodiments, an audio device 980 (e.g., a speaker 981 connectedto a source of audio signal 982) is included in trap system 900 with thesticky surface 910 to attract arthropods. In some embodiments, audiosource 980 provides sounds that attract certain arthropods. In someembodiments, side light source 920 and or front light source 921 (e.g.,one or more various different colors of LEDs such as infrared, red,orange, yellow, green, blue, and/or ultraviolet colors, in someembodiments) is chosen and illuminated, e.g., at night, to attractcertain arthropods to the trap, as well as to provide illumination fortaking the image. In some embodiments, sticky sheets 910 meet colorrequirements that are attractive to certain species. In someembodiments, sticky sheets 910 are made to withstand elements of anoutdoor environment (e.g., sheets having sunlight resistance andcold/heat resistance).

In some embodiments, the methods and apparatus of the present inventionare also used to analyze images of arthropods whose cuticles (externalsurface) have been tagged with diagnostic markers (“taggants”) that haveaffinities for a specific cuticle component such as hydrocarbons (see,e.g., Bergman, D. K., J. W. Dillwith, R. K. Campbell, and R. D.Eidenbary, 1990. Cuticular hydrocarbons of the Russian wheat aphid.Southewestern Entomologist. 15(2): 91-99), waxes and lipids (see Lockey,K. 1988. Lipids on the insect cuticle: Origin, composition and function.Comp. Biochem. Physiol. 89B(4): 595-645., 1988). In various embodiments,the markers are fluorescent materials, other materials (for example,tissue stains or afterglow phosphors) or radioactive materials. Chemicalor topographical variations of the arthropod cuticle among species areused to discriminate insect populations. For example, there arevariations in cuticular hydrocarbons between different Russianwheat-aphid populations (Bergman et al. 1990, cited above). An extensivereview of literature on markers to tag insects is provided by Southwood,1978 (Southwood, T. R. E. 1978. Ecological Methods. Chapman and Hall.London. 524 p.). Taggants with affinity for specific cuticle componentscan be applied to trapped arthropods or arthropods placed on a detectionsurface. Digital images are taken and analyzed for specific spectra fromthe tagged cuticle components. One use for this is in forensicentomology, i.e., the identification of insects and other arthropodsthat infest decomposing remains, in order to aid criminalinvestigations.

In some embodiments, the method of the present invention is also used toexamine digital pictures of manually-prepared tissue sections (e.g.,slices of arthropods or other organisms, including, in some embodiments,sections of human or other mammalian, avian, piscine, reptilian, orother animal or plant tissues) that have been labeled withmonoclonal-antibody or DNA-specific-sequence taggants using well-knownlabeling techniques such as described in the above references. Forexample, a tissue sample is obtained and prepared and a selectivetaggant is applied (such as one or more different tissue stains, and/ormonoclonal-antibody or DNA-specific-sequence taggant), and a digitalphotograph is taken. In some embodiments, a microscope is used to obtaina greatly enlarged image of suitable resolution. The image-analysismethods described herein are then used to locate and isolate areas ofinterest in the image (in some embodiments, a human-user interface isprovided to enhance the identification of areas of interest), and themethods of the invention then utilize, for example, color histograms orcolor patterns of each area of interest, in order to identify the typeof organism, or to identify an indication of some pathology such ascancer or bacterial infection.

As shown in FIG. 9, some embodiments further include an optional taggantstation 986, for example including a surface 988 across which thearthropods would be expected to walk, and a funnel 987 leading to blacksurface 914. An arthropod walking across or passing through taggantstation 986 would pick up taggant on some portion of its body, forexample, on its feet, much like a child walking through mud. In someembodiments, the taggant is specifically targeted to selectively attach(or selectively not attach) to one type of arthropod (a targetedtaggant), while other embodiments use a taggant that non-selectivelyattaches to any arthropod passing through (a non-selective taggant). Insome embodiments, the LEDs 920 near the black surface 914 include LEDsthat emit ultraviolet or “blacklight” such as are available from NichiaAmerica Corporation. In some embodiments, a tagged arthropod, uponexiting taggant station 986 would end up stuck to sticky black surface914, and photographed. For example, in some embodiments, a first,normal-light, image is obtained using one set of LEDs 920 or ambientlight, and a second, blacklight, image is obtained of the same sceneusing UV emitting ones of the LEDs 920, and showing, for example,fluorescently re-emitted light from taggant on the tips of the feet ofthe arthropod. Analysis of the two images is then done in a combinedfashion, using features obtained of the arthropod from the first imageand from the second image, for example obtaining colors or colorpatterns from the first image and other information such as outlineinformation, e.g., the positions of the ends of the limbs, from thesecond image, and then performing the recognition methods of the presentinvention on all of the information.

Thus, in some embodiments, the present invention includes acquiring twoor more images of the same scene (using either the same or differentimagers), and providing different lighting (such as differentwavelengths (e.g., UV, visible, and/or infrared), differentpolarizations or filters, and/or different source directions) for eachof the plurality of images. In some such embodiments, taggants are used,for example to provide fluorescence for the UV image, while in otherembodiments, no taggant is used and the two images obtain differentcolor, fluorescence, or polarization information of the specimens intheir natural state. For example, a first image can be obtained innormal light of a red-green-blue spectrum (e.g., using red, green andblue LEDs for illumination, and an RGB imager), and then a second imageof the same scene is obtained using the same RGB imager, but with onlyUV LEDs providing illumination and the imager obtaining light from thefluorescing specimen or the taggants attached thereto. In someembodiments, further images are also obtained, e.g., using differentpolarizing filters. Since the same imager in the same position is used,corresponding pixel locations from the different images providedifferent information about the same area of each specimen. Theadditional images are, in some embodiments, treated as additional colorvalues, such that a first hue-saturation-intensity set of values foreach pixel is obtained from the RGB visible-light image, and a secondhue-saturation-intensity set of values for each pixel is obtained froman UV-light/fluorescing or phosphor-afterglow image. In someembodiments, histograms or color patterns of these additional colorsprovide additional inputs to the identification portions of theobject-recognition portions of the method and apparatus for theinvention described herein.

In some embodiments, the targeted taggant 989 is sprayed on from anaerosol can, and when on surface 988, will selectively either stick ornot stick to a particular set of arthropod types. At the targetedtaggant station 988, for example, only a small set of species, or evenone specific species or one sex of arthropod would pick up some of thetargeted taggant. Targeted taggants include chemical markers, taggants,radio-isotopes, afterglow phosphors, and substances withphoto/thermal-chemical effects such as fluorescence, to which areattached antibodies or DNA snippets or other chemical keys specific tothe set of species, or one specific species or one sex of arthropod ofinterest. For example, arthropods have cuticles that have wax coatings.Different arthropods have different waxes. Antibodies exist that stickto certain of these waxes and not to others. The term “taggant” is usedto label the technology associated with the chemical tagging of marks,inks or toners or similar substances such that “tagged” objects can bedistinguished from “untagged” objects. In some embodiments, the tagganteffect may be readily observable such as the application of materialsthat change colors with slight temperature changes or when viewed atvarying angles or when illuminated by “black light” or flashed with ashort pulse of bright light. Taggants can involve a number ofphoto-chemical effects such as; absorbing energy at one wavelength andemitting energy at another, absorbing energy at particular wave-lengths,temporal effects when illuminated with pulsed energy, etc. Taggants canalso include radio-isotopes that can be detected with detectors forradioactivity such as Geiger counters.

For example, in some embodiments, a species-specific fluorescent taggant989 is placed on surface 988 of taggant station 986. When an individualof that specific species walks across surface, it picks up some of thetaggant 989 on its feet, travels through optional funnel 987 and becomesensnared on sticky black surface 914. Ultraviolet LEDS 920 emit UV lightthat is absorbed by the taggant and re-emitted at a longer wavelength(such as yellow or green) that is readily detected by imager 310.Individuals of other species would not pick up the taggant (for example,because their different waxes do not have an affinity for the taggant),and if these untagged individuals end up on black surface 914, theywould not fluoresce. This difference provides another distinguishingfeature that is used by the software to distinguish and identifyindividuals from a specified set of arthropods (such as one species).

In some embodiments, the taggant station 986 is sold as a consumable.These preconfigured taggant stations 986 can then be sold to users formore specific identification uses.

In some embodiments, a preconditioned sheet 910 includes the taggantstation 988 as one part of the sheet, such that, for example, thetaggant surface 988 is surrounded by black surface 914. Thesepreconfigured taggant sheets 910 can then be sold to users for morespecific identification uses.

In some embodiments, trap system 900 also includes one or more colorfilters 932 and/or polarizing filters 931 to condition the light forobtaining higher-quality or better-contrast images, and a lens system940 and imaging electronics (such as a CCD or CMOS detector array andthe driving circuitry) is suitable resolution to obtain images withsufficient quality for the automatic image processing of the presentinvention.

FIG. 10A is a representation of a calibration surface 915 used in someembodiments of the invention. In some embodiments, calibration surface915 is included as a small portion of an overall collection and imagingsurface 900 as shown in FIG. 9. Some embodiments include a grid ofsquares, each having a different color, hue, saturation, and/orintensities 916, 917, 918. Since the predetermined colors are of knownvalues, an image of patch 915 can be used to calibrate the colors, hues,saturations, and/or intensities of an associated collection image ofarthropods. Some embodiments include a printed card containing astandard or particular combination of hue and saturation for each of thefollowing colors: red, blue and green, or yellow, magenta, and cyan. Animage of this card (as part of a field image of collected arthropods) isused and compared to a standard to adjust the color settings on variousimaging devices such as scanners, digital video cameras and digitalstill-frame cameras, so that different imaging devices and differentlighting conditions can be calibrated to produce arthropod imagesequivalent in color.

FIG. 10B is a graph of an example calibration function 1010 used in someembodiments of the invention. For example, for certain imaging hardwareunder certain lighting conditions, a curve 1011 is derived from imageinformation correlated to patch 915. The correction function is thenderived to change the pixel information for the entire image to astandard (e.g., linear) curve 1012. In addition to the being able toreproduce identical pigmentation from card to card, the card is, in someembodiments, printed on paper, plastic or other material, where thepigments are uniformly applied, with reflective glare minimized, and thetexture of the material's surface minimized relative to the spatialresolution of the cameras. In some cases, depending on the resolution ormagnification of the imaging devices, the paper for the calibration cardis of a quality that does not have detectable strands or chips of woodfibers. In some embodiments, the calibration card is not limited to justthe visible portion of the light spectrum. In some embodiments, system1070 (See FIG. 10H described below) uses imaging devices 310 (andcalibration cards 915) that obtain and/or calibrate image informationusing light beyond the visible spectrum to look for distinct colorpatterns of arthropods in the near ultraviolet or near infrared, or fortagged or fluorescent molecular markers on the arthropod's surfaces. Insome embodiments, LEDs 920 and/or 921 (see FIG. 9) emit light that is atleast partially in the ultraviolet or infrared spectra.

FIG. 10C shows a collecting device 1020 adapted to vacuum devices tosample insects. In some embodiments, device 1020 includes an inletopening 1021 through which air is drawn and large enough to admitarthropods of interest and optionally small enough to keep out largeranimals such as bees or hummingbirds, a chamber 1029, a perforatedsubstrate 910 on an inner surface through which air is drawn intomanifold 1022 and vacuum passage 1023 connected to vacuum pump or fan1024. In some embodiments, perforated substrate 910 has holes 1025 and asticky surface to hold the collected arthropods, while in otherembodiments, the vacuum alone is enough to hold the collected arthropodslong enough to obtain the desired image using imaging device 310.

FIG. 10D is a perspective view of a sample-cleaning system 1030 used insome embodiments. Some embodiments include sets of sieves (e.g., tiltedsieve 1031 with large openings, tilted sieve 1033 with medium openings,and tilted sieve 1035 with fine openings) and/or blower(s) 1037 toseparate arthropods from non-arthropod material, and/or to separatedifferent types of arthropods from one another. In some embodiments, thesource material is deposited into the open top. The size- and/orweight-sorted arthropods and other objects are then obtained fromspigots 1032, 1034, 1036 and 1038.

FIG. 10E is a perspective view of a sample-processing unit 1040 used insome embodiments. In some embodiments, sample processing unit 1040includes a vessel or container 1042 (with a closable opening 1041) inwhich to place samples of arthropods prior to acquiring their image. Insome embodiments, container 1042 includes a means of immobilizing orkilling the arthropods. Immobilizing methods include using ether orethyl acetate, or cold temperature. In some embodiments, container 1042also contains a plaster-of-Paris (hemi-hydrated calcium sulfate)substrate 1043 to hold or absorb any volatile liquids that are used tokill or immobilize the arthropods. There are several variations(different embodiments) for immobilizing the arthropods. In someembodiments, container 1042 has a separate compartment for solids thatwould prevent arthropod mobility. In some embodiments, ammoniumcarbonate, ice, and/or dry ice, are placed in this compartment to kill,render immobile, or knock out arthropods. In some embodiments, container1042 could also be fitted with one or more regulator valves 1044 thatcan be screwed onto a CO₂ cartridge 1045. A controlled quantity of CO₂is released into the container to render immobile or knock out thearthropods.

FIG. 10F is a perspective view of a set of scanner lids 1050 used insome embodiments. In some embodiments, set 1050 includes scanner lids invarious standard colors (e.g., lid 1051 with a black background, lid1051 with a blue background, and lid 1053 with a yellow background,and/or a lid with a white background) are provided to cover the scannersurface, if such is used to obtain images of the arthropods. Forexample, a sample of arthropods are deposited on the glass scannersurface and covered with one or another of the set of lids 1050 toobtain one or more images with different backgrounds to improvecontrast. In some embodiments, set of lids 1050 are constructed out of apaper product or plastic with a matte surface to reflect light without aspecular (mirror-like) reflection. Some embodiments include severaloptimized colors to allow for the selection of a background color thatmaximizes the difference in hue and saturation between the expectedinsects and their background. Studies with scanners indicate that, insome embodiments, a lid about five centimeters high may be the optimumheight.

FIG. 10G is a block diagram of an example on-linearthropod-identification service 1070. In some embodiments, one or moreusers 87 upload (transmit) one or more images 1071 of unknown arthropodsor other objects to a commercial and/or non-profit website hosted bysystem 320 (see FIG. 3). In some embodiments, the images are optionallyaccompanied by other information such as the place, environment and timeof the collection, and optionally including billing information such ascredit-card data (to pay for the identification service) that is enteredthrough a secure interface and stored to database 1079. In system 320,automated software analyzes and classifies the objects found, andreturns an identification, and optionally also sends other relevantinformation (such as control methods and substances, and/or imageinformation to help the user confirm the machine-made classification) onthe identified species. In other embodiments, images of unknowns aresent to the automated identification service provided by system 320 viamail, email or facsimile machine (fax). In some embodiments, the sourceimage needs to conform to certain image formats, standardized lightingand camera settings, pixel resolutions, etc. In some embodiments, theimage is pre-processed to obtain condensed image information, such ashistogram and silhouette information, which is transmitted and analyzedby proprietary software (using reference database 200 or 1060) at acentralized location, which returns the identification and/or otherinformation. In some embodiments, the system 320 stores theidentifications made and the information such as place, environment andtime of the collection, into a centralized results-and-analysis database250, where various further analysis and data-aggregation functions canbe performed.

In some embodiments, the following method is used with FIG. 10G:establishing a network connection, transmitting image informationwherein the image information includes image information data regardingone or more arthropods, analyzing the image information and generatingclassification information regarding identified arthropods, andreturning the classification information. In other embodiments, imagesof other organisms or their parts (e.g., plants, fish, feathers frombirds, shed skins of snakes, X-rays of human patients or other pathologyor microscopy images, etc.), or of non-living items (e.g., rocks,crystals, fossils, antiques, or human-made items) other than arthropodsare transmitted, analyzed, and the identification or classificationreturned. In some embodiments, payment information is solicited from theuser 87, and collected into database 1079, in order to charge the userfor the service(s) provided. In some embodiments, different paymentamounts are requested based on how much classification and analysisinformation is requested (e.g., just an automated classification mighthave a low cost, or additionally a human confirmation of theidentification at a higher cost, and/or information as to controlmethods, or image data returned might have different cost rates.)

FIG. 10H shows a diagram of an example reference database structure 1060for key arthropods. Some embodiments include one or more differentreference databases of important arthropods for classification. In someembodiments, each database is tailored for particular agricultural crops(e.g., for field use) and/or commodities (e.g., for use in grainelevators or other commodity-storage facilities), or other specializedor identified environments. In some embodiments, each record 1061includes a plurality of fields, for example species filed 1062, genusfield 1063, silhouette data field 1064, hue and saturation data field1065, etc. In some embodiments, each record 1061 further includes a datafield 1068 describing methods that can be used to control thatparticular arthropod, and data field 1069 having image data for thatparticular arthropod, for example in GIF or JPEG format, or a pointer(such as a URL) to a GIF or JPEG image.

In some embodiments, as shown in the lower portion of FIG. 10H, eachdatabase 1060 contains a plurality of records 1091, 1092, etc., thatinclude a sufficient representation of the variation in appearance(e.g., record 1091 that includes a plurality of silhouette fields 1095,hue-saturation histogram fields 1096, and other identification fields1097 for the arthropod type identified in species field 1093 and genusfield 1094) of the important and common arthropods for a particular cropor environment. In some embodiments, each database 1060 includesinformation 1098 as to control methods and compounds (e.g.,insecticides) for the identified arthropods, and/or a set of arthropodimages 1099 that provide interested parties with images of arthropodsfrom the image database. In some embodiments, rather than holding theimages and other auxiliary information directly, database 1060 includespointers to internet web pages having the desired information. Thishelps the user by correlating the identification made by the system 320to images, control methods, or other information about particulararthropods. Images are useful for researchers and educators. Someembodiments provide access to this database information, or to certainparts thereof, as part of a business method implemented to be available(optionally for a fee) over the internet.

FIG. 10I and FIG. 10J show first and second portions of a diagram of anexample reference database structure record 1080 showing exemplary data,typical of some embodiments, for one key arthropod, a particular weevil.The comments following the double slash marks “//” in each field aretypically not included in each record but are shown here for clarity. Insome embodiments, each record 1080 includes data such as, in FIG. 10I:

-   FIELD 01—CLASS STRING (e.g., “WEEVIL”);-   FIELD 02—SUBCLASS STRING (e.g., “WEEVIL SIDE VIEW”);-   FIELD 03—CLASS NUMBER (e.g., “1”);-   FIELD 04—SUBCLASS NUMBER (e.g., “1”);-   FIELD 05—AREA OF ARTHROPOD (e.g., “1292”);-   FIELD 06—PERIMETER (e.g., “202”);-   FIELD 07—LENGTH (e.g., “57.922”);-   FIELD 08—WIDTH (e.g., “34.461”);-   FIELD 09—CIRCULAR MATCH FEATURE (e.g., “0.398”);-   FIELD 10—RECTANGULAR MATCH FEATURE (e.g., “0.647”);-   FIELD 11—ELONGATION (MAJOR TO MINOR AXIS) (e.g., “1.681”);-   FIELD 12—TWELVE VALUES OF THE SHAPE HISTOGRAM (e.g., twelve bins);-   FIELD 13—AVERAGE GRAY LEVEL (e.g., “—66”);-   FIELD 14—SIXTY-FOUR VALUES OF THE INTENSITY HISTOGRAM; and in FIG.    10J:-   FIELD 15-32 by 32 COLOR-SATURATION MATRIX (e.g., typically mostly    zeros with groups of peaks corresponding to the hues and saturations    of the main colors); and-   FIELD 16—COLLECTION ID (e.g., character string such as “08032 CBW    0000”).

FIG. 10K and FIG. 10L show first and second portions of a diagram of anexample reference statistical-feature database definition 1081 for keyarthropods. Each field in the database definition 1081 corresponds tothe same field in database structure 1080, and provides a furtherexplanation of those fields. In some embodiments, database structure1080 is used in the process explained in FIG. 8A above, and inparticular in statistical classifier 850 and syntacticclassifier/silhouette matcher 860, and/or in the “match outlinegeometry” block 130 of FIG. 1.

FIG. 10M and FIG. 10N show first and second portions of an exampledefinition of reference color-silhouette database 1082 for keyarthropods. In some embodiments, database 1082 is used in the “matchcolor geometry” block 140 of FIG. 1.

FIG. 11 is a flowchart of a method 1100 according to some embodiments ofthe invention. At block 99, the image is taken, acquired, or input tothe classification computer. In some embodiments, block 1200 includesthe operation of detecting arthropods using color and/or luminescence1240, the operation of calculating one or more adaptive thresholds 1250,and combined segmentation logic for detection 1260, each of which isdescribed further in regard to FIG. 12 below. Block 1500 includes theoperation of creating a 2D histogram, which is described further inregard to FIG. 15 below. Block 1600 includes the operation of comparingthe 2D histogram, which is described further in regard to FIG. 16 below.

Block 1700, which is described further in regard to FIG. 17 below,includes the arthropod-classification operations of applying a modifiedKNN 1800, which is described further in regard to FIG. 18 below,evaluating the KNN result 1900, which is described further in regard toFIG. 19 below, and applying a syntactic classifier 2000, which isdescribed further in regard to FIG. 20 below. Block 1199 represents theoperation of outputting one or more candidates, and optionally alsooutputting a confidence for each candidate.

FIG. 12 is a flowchart of a method 1200 according to some embodiments ofthe invention. In some embodiments, method 1200 includes the function ofinputting 1210 the image of interest (the one to be analyzed) as well asan earlier image of the same substrate of a representation of thebackground image (e.g., a yellow image if the original substrate wereyellow). The next function of creating 1220 intensity, hue, andsaturation images based on the image of interest and the earlier orbackground image, as well as on formulae 1230 wherein, in someembodiments for each pixel in each image,

-   formula 1231: INTENSITY=0.299×RED+0.587×GREEN+0.114×BLUE-   formula 1232: CR=0.701×RED+0.587×GREEN+0.114×BLUE-   formula 1233: CB=−0.299×RED+0.587×GREEN+0.886×BLUE-   formula 1234: SATURATION=SQUARE ROOT(CR SQUARED+CB SQUARED)-   formula 1235: HUE=ARCTAN(CR/CB).

Next, the function of creating 1240 difference images from the threesets (intensity, hue, and saturation) of current and background imagesis performed. Next, the function of creating and applying 1250 (one suchembodiment is further described in FIG. 13 below) adaptive thresholds(i.e., function 1251 of applying adaptive threshold to the intensitydifference image, function 1252 of applying adaptive threshold to thesaturation difference image, function 1253 of applying adaptivethreshold to the hue difference image). Next, the function of applying1260 (one such embodiment is further described in FIG. 14 below)combined-segmentation logic is performed. Then, the function of applying1270 connected-components analysis is performed to create alabeled-detection image (i.e., for each pixel, examining neighboringpixels in each of a plurality of directions to determine which pixelsare “connected” (i.e., form part of the same detected object—called a“detection”—in the image). In some embodiments, the background pixelsare set to a value (e.g., zero) and the other pixels (e.g., possiblearthropods) are set to another value (e.g., 255). Then the first “255”pixel (e.g., the left-most and top-most) is processed (e.g., its valueis set to one, and its neighbor pixels and their neighbors, if 255, arealso set to one. Then the next “255” pixel (e.g., the left-most andtop-most of the remaining pixels) is processed (e.g., its value is setto two, and its neighbor pixels and their neighbors, if 255, are alsoset to two. Then the next “255” pixel (e.g., the left-most and top-mostof the remaining pixels) is processed (e.g., its value is set to three,and its neighbor pixels and their neighbors, if 255, are also set tothree, and so on. In some embodiments, each pixel of thelabeled-detection image can be represented by a 16-bit word. In thisway, up to 65,535 groups of pixels can be identified as “detections” orseparate detected objects. In other embodiments, other values can beused, depending on the number of objects to be identified.

FIG. 13 is a flowchart of a method 1250 according to some embodiments ofthe invention. At block 1310, the function of method 1250 is started foreach of the intensity, hue, and saturation images. At block 1320, thefunction of creating a histogram with the absolute value of thedifference between the entire image (or substantially the entire image)of interest and the corresponding prior or background image isperformed. For example, the histogram might have 256 “bins;” one bin(counter value) for each possible absolute value of the difference valuebetween corresponding pixel values of the two images. Bin 0 is a counterthat would have the number of pixels that have zero difference (a countof the pixels that have the same value in the prior image and the imageof interest); bin 1 would have the number of pixels with a difference ofplus or minus 1, bin 2 would count the pixels that have a difference ofplus or minus 2, and so on. At block 1322, the function of setting thethreshold to a default value, and selecting as a threshold bin if 15% ofthe pixels have a larger difference (e.g., taking as an initialassumption that 15% or fewer pixels will be of an arthropod or otherobject), is performed. At block 1324, the function of finding the binwith the peak value within the first 30 bins (the bins that count thezero difference to the twenty-nine difference) is performed. At block1326, the function of calculating the positive standard deviation aboutthe peak bin (e.g., for a standard deviation of one and four, todetermine a search range) is performed. At block 1328, the function ofcalculating the minimum bin size to continue the search (e.g., in someembodiments, a minimum bin size or value would include at least 0.15% ofthe total pixels) is performed. At block 1330, a branch is made based onwhether there is a bin of the minimum size between the peak and ½standard deviation to its right (bins with larger differences). If yes,then at block 1332 the value in this “empty” bin is used to set thesearch threshold, and at block 1334, if the search threshold is greaterthan the default threshold, then the default threshold is used; else thesearch threshold is used. If at branch 1330, there is no bin of minimumsize between the peak and ½ standard deviation to its right, then block1340 is performed, where the search region is set to between a bin at ½standard deviation and a bin at some maximum standard deviation (e.g., avalue between two and four standard deviations) to the right of peak.The threshold is set when the function encounters two consecutivelylarger-valued bins, or a minimum-sized bin. If at branch 1350, thethreshold was found before the maximum-standard deviation bin, then thesearch threshold is set at that bin, and the process goes to block 1334.If at branch 1350, the threshold was not found before themaximum-standard-deviation bin, then the search threshold is set at themaximum-standard-deviation bin, and the process goes to block 1334. Thisprocess then iterates until an appropriate threshold is found (e.g.,because sufficient convergence is seen), for each of intensity, hue, andsaturation difference images.

FIG. 14 is a flowchart of a method 1260 according to some embodiments ofthe invention. In some embodiments, method 1260 is used for theintensity difference image with entry at block 1410, for the saturationdifference image with entry at block 1412, and for the hue differenceimage with entry at block 1414. Block 1420 represents a common launchpoint for each image pixel of each type. If, at branch block 1430, thepixel is determined to be a “bright” pixel (wherein theintensity>threshold value), then at block 1432 that pixel is marked as apotential arthropod pixel (or as “not background” if other thanarthropods are being examined). Else, if at block 1430, the pixel is not“bright” then if at branch block 1440, the pixel is determined to be“too dark for shadow” pixel (wherein the intensity<−threshold value and<−40), then at block 1432 that pixel is marked as a potential arthropodpixel. Else, if at block 1440, the pixel is not “too dark for shadow”then if at branch block 1450, the pixel is determined to be “dark asshadow, but not” pixel (wherein there is a change in hue or saturation),then at block 1432 that pixel is marked as a potential arthropod pixel.Else, if at block 1450, the pixel is not “dark as shadow, but not” thenif at branch block 1460, the pixel is determined to have “a change inhue and more than minimum saturation” (wherein the hue>threshold valueor hue<−threshold), then at block 1432 that pixel is marked as apotential arthropod pixel. Else, if at block 1460, the pixel is not“change in hue and more than minimum saturation” then if at branch block1470, the pixel is determined to have a “change in saturation” (whereinthe saturation>threshold value or saturation<−threshold), then at block1432 that pixel is marked as a potential arthropod pixel. Else at block1472 the pixel is marked as a “background” pixel.

FIG. 15 is a flowchart of a method 1500 for creation of a 2D colorhue-versus-saturation histogram for arthropod or other objectclassification, according to some embodiments of the invention. In someembodiments, method 1500 starts at block 1510 with input of the originalimage (or the original as color-corrected by a method using FIG. 10A andFIG. 10B) and a labeled-detection image. At block 1520, for eachdetected object (“detection”) in the labeled-detection image, the methodpasses control to block 1530; where for each pixel of the detectedobject the method goes to block 1532. At block 1532, some embodiments ofthe method include calculating the pixel's CR and CB value from its RGBvalues in the corresponding original image pixel using the formulae ofblock 1534: CR=0.701R−0.587G+0.114B, and CB=−0.299R−0.587G+0.886B. Atblock 1536, depending on color resolution desired, the method optionallyincludes scaling the CR/CB values. In some embodiments, the default isto reduce the values from an 8-bit value down to a 5-bit value bydividing by 8 (or shifting the value right by three bits todelete/ignore those three low-order bits). The result is then a 32×32rather than a 256×256 histogram. At block 1538, some embodiments of themethod include incrementing by one the histogram's bin whose row andcolumn correspond to the pixel's CR and CB values. If at branch block1540, there are more pixels in the detection to process, then controlreturns to block 1530 for the next pixel in this detection. Else, if atbranch block 1540, there are no more pixels in this detection toprocess, then some embodiments of the method include dividing each binof the histogram by the detection's area (by the number of pixels inthis object). Each value will then be the fraction of the detection thathas that bin's combination of hue and saturation. Then, if at branchblock 1560, there are more detections (detected objects) in the image toprocess, then control returns to block 1520 for the next detection inthis image. Else, at block 1570, the histograms are complete, and anidentification process (such as described in FIG. 16-20) is started.

FIG. 16 is a flowchart of a method 1600 for comparing a 2D colorhue/saturation histogram for an unknown with the histogram of areference, according to some embodiments of the invention. In someembodiments, method 1600 starts at block 1610 with input of areference-specimen file containing features and color histogram for eachreference specimen (i.e., from a database of previously analyzed andidentified arthropod specimens). In some embodiments, at block 1620, themethod reads histograms of the “knowns” (known specimens) from thereference-specimen file. In some embodiments, at block 1630, a colorhistogram of the unknown detection is generated (or, in someembodiments, is obtained as an output of method 1500 of FIG. 15). Atblock 1640, one of the reference histograms is selected for comparison.At block 1650, some embodiments of the method include initializing anoverall difference in an overlap variable to zero. At block 1660, themethod for each corresponding bin of histograms, goes to block 1662. Atblock 1662, some embodiments of the method include calculating theabsolute difference between the two bins. At block 1664, someembodiments of the method include adding the bin difference to theoverall difference in overlap. If, at block 1664, this is not the lastbin, then the method goes to block 1660 to process the next bin; else atblock 1670, some embodiments of the method include dividing the overalldifference by 2.0 to get a decimal fraction of non-overlap. At endingblock 1680, the results in the normalized feature difference are next tobe used by a KNN (K-nearest-neighbor) classifier, such as described inFIG. 17, in some embodiments.

FIG. 17 is a flowchart of a method 1700 having a K-nearest-neighborclassifier (statistical-feature classifier) approach to arthropodclassification, according to some embodiments of the invention. Block1710 represents the input of one or more reference feature sets,including, in some embodiments, a sample mean and standard deviation foreach feature of each species, and block 1712 represents the input of theunknown's feature set, these inputs going to the starting point of block1720. At block 1730, some embodiments of the method include evaluatingwhether a classification decision of KNN classifier produced a goodmatch. If, at block 1732, this is a good match, then at block 1734, thisclassification is output or stored in a database of generatedidentifications or classifications. Else, from branch block 1736 if themethod is not to do silhouette/color sample matching, then at block1738, an output or database entry of “other” classification isindicated, i.e., the unknown is indicated as not represented inreference set. Else, at block 1740, a determination is made of whetherthe silhouette and color-reference pixel(s) match (e.g., in someembodiments, as in FIG. 20 described below), in order to confirm orreject the best match of the KNN classifier. At branch block 1750, ifthe silhouette/color match does confirm the best statistical match, thenat block 1755, some embodiments of the method include outputtingidentification or class of the KNN classifier (or storing it into adatabase); else the match is not confirmed, and at block 1760, someembodiments of the method include finding the best matches for eachprototype silhouette. Then, at block 1770, some embodiments of themethod include assigning the class of the best silhouette match to eachportion of the unknown not previously explained by a prototypesilhouette. This will explain occlusions. If the best match for an areahas an insufficient number of matching pixels, then that area belongs tothe class “other.”

FIG. 18 is a flowchart of a method 1800 providing a modified KNNclassifier for arthropod identification, according to some embodimentsof the invention. In some embodiments, method 1800 starts at block 1810for each known of one or more reference feature sets. At block 1820,some embodiments of the method include setting a sum-of-squares variableto zero and goes to block 1822. From block 1822 for each selectedfeature, the method goes to block 1830. At block 1830, some embodimentsof the method include taking a difference between the feature of theknown and the feature of the unknown. At block 1832, some embodiments ofthe method include normalizing the difference by dividing the differenceby the known's feature value and then squaring the quotient. At block1840, some embodiments of the method include adding the result to asum-of-squares variable. If from block 1842 there are more features, themethod returns to block 1822; else the method goes to block 1850. Atblock 1850, some embodiments of the method include assigning a Euclideandistance for this feature as the square root of the sum of squares. Atblock 1860, if this Euclidean distance is among the K nearest distances,then some embodiments of the method include performing an insertion sortof this Euclidean distance into the list of the nearest Euclideandistances. If at branch block 1862, there are more knowns, then themethod returns to block 1810; else at block 1870, some embodiments ofthe method include assigning the classification of the majority voteamong the K nearest knowns to the unknown. At block 1880, someembodiments of the method include evaluating whether the nearest matchof this class is a good match as describe below for FIG. 19, (and/orassigns a confidence factor to the match).

FIG. 19 is a flowchart of a method 1900 that provides an evaluation ofwhether KNN classifier found a good match, according to some embodimentsof the invention. At block 1910, some embodiments of the method includegetting the feature data for the unknown and the known nearest neighbor(KNN) of the classification. At branch block 1912, if the method is touse statistical methods, control goes to block 1920; else control goesto block 1930. From block 1920, for each feature, the method goes toblock 1922. At block 1922, some embodiments of the method includeperforming a Grubbs' test for a statistical outlier (Grubbs' testcalculates a ratio called Z, where Z is equal to the difference betweenthe unknown's feature value and the mean value of the referencespecimens of the class that best matches the unknown, divided by thestandard deviation among the reference specimens of the best matchingclass. The mean and standard deviation also include the unknown. If Zexceeds a critical value for a given confidence level, the decision ofthe 1NN classifier can be rejected.). At branch block 1924, if thefeature value is an outlier, then control passes to block 1940; elsecontrol passes to branch block 1926, where if more features are to beexamined, control returns to block 1920. Else, if there are no morefeatures, control passes to block 1950 and some embodiments of themethod include measuring overall “goodness of fit” with a chi-squaredtest or other additional multivariate outlier test such as theMahlanobis distance-squared test. Then, at branch block 1952, if the fitis good, control passes to block 1980 and some embodiments of the methodinclude outputting the classification; but if the fit is poor, thencontrol passes to block 1970. At branch block 1970 (and at branch block1940), if the identification needs to be confirmed, then control passesto block 1942, and some embodiments of the method include passing theclassification to the silhouette/color-matching classifier of FIG. 20;else the method passes control to block 1944 and some embodiments of themethod include outputting a classification of “other.”

If, from block 1912, it is decided not to use statistical methods,control passes to block 1930. At block 1930, for each feature, someembodiments of the method include calculating a percentage difference as((unknown's value−known's value)*100/known's value) and control passesto block 1932. At branch block 1932, if the percentage differenceexceeds a user-provided threshold (where the default threshold is 100%),then control passes to block 1940 described above; else control passesto branch block 1934, where if more features are to be examined, controlreturns to block 1930. Else, if there are no more features, controlpasses to block 1960 and the method calculates an average percentdifference among the features and control passes to block 1962. Then, atbranch block 1962, if the fit is good, control passes to block 1980 andthe classification is output; but if the fit is poor, then controlpasses to block 1970 described above.

FIG. 20 is a flowchart of a syntactic classifier method 2000 thatprovides silhouette and/or color-reference-pixel matching according tosome embodiments of the invention. In some embodiments, silhouettematching finds a “center-of-mass” point of the unknown silhouette thatis then placed in the position of the “center-of-mass” point of thereference silhouette. The unknown silhouette is then matched to thereference, rotating (e.g., using a linear transform) incrementallybetween each matching operation (and optionally translating thecenter-of-mass point) until the orientations of the unknown silhouetteand the reference silhouette most closely match. In some embodiments,color-reference-pixel matching then takes a known starting point (e.g.,a point at the head of the arthropod) and examines a pixel at apredetermined X and Y offset (or equivalently at a predetermined angleand distance) to check for a match of the hue and/or saturation of thatpixel or area on the unknown to the hue/saturation of the correspondingpixel or area of the reference image (i.e., in some embodiments, thereference database stores characteristic “spotprints” of the referenceimages, wherein at each pixel of a characteristic set of one or moregiven X and Y offsets, arthropods can be distinguished by the hue andsaturation found there). Thus, rather than matching the entire colorpattern, a relatively small subset of important or distinguishingoffsets, hues, and saturations are matched. Some embodiments combine thematching of silhouette and of hue/saturation spots after each rotationand/or translation of the unknown silhouette (or equivalently, in otherembodiments, the prototype silhouette is rotated).

In some embodiments of method 2000, block 2010 includes reading from areference file a set of one or more reference “spotprints” (eachspotprint having a prototype silhouette and a set of characteristiccolor-reference pixels (CRP), e.g., in some embodiments, each CRPspecifying X offset, Y offset, hue, and saturation). At block 2020, themethod generates a silhouette of the detected unknown object(“detection”). From block 2030, for each prototype silhouette, themethod starts by translating the silhouette of the unknown detection sothe center of the detection silhouette overlaps the center of theprototype silhouette and passes control to block 2040. For eachpermutation of rotation and translation of the prototype silhouette, atblock 2040, the method passes control to block 2050. At block 2050, someembodiments of the method include calculating percentage of silhouettepixels that overlap (in some embodiments, to within some giventolerance) the unknown's silhouette and control is passed to block 2052.At block 2052, some embodiments of the method include calculatingpercentage of reference-color pixels that match hue and saturation ofcorresponding pixels in original color image and control is passed toblock 2054. At block 2054, some embodiments of the method include savingthis match if the number of matching pixels is good and if it is amongthe n best matches found thus far and control is passed to block 2056.At branch block 2056, if more orientations are to be tested, thencontrol returns to block 2040; else control passes to block 2058. Atbranch block 2058, if more prototype silhouettes are to be tested, thencontrol returns to block 2030; else control passes to block 2060. Atblock 2060, starting with the best acceptable match, some embodiments ofthe method include assigning the class of that best match to theunknown. If a large portion of the unknown is not explained by the knownsilhouette, some embodiments of the method include assigning thatportion to the best acceptable match that covers it. Some embodimentsrepeat block 2060 until all portions of detection are classified. If theunknown or portions of it are not matched then some embodiments of themethod include assigning that unknown or portion thereof toclassification “other.”

FIG. 21 shows a portion of YCbCr space where the luminosity, Y, is keptat a constant gray level of 128 across the entire space. The x-axis orcolumns represent the Cb axis where values range from −127 on the leftmost portion of the image to 128 on the right side of the image (aslabeled). The y-axis or rows represent the Cr axis where the valuesrange from −127 at the top of the image to 128 at the bottom of theimage. Note that the hue changes as an angle around the center or origin(0,0) of the YCbCr color space and the color becomes more saturated thefurther you are from the center or origin of the YCbCr color space. Thecentral pixel of the image or origin of the YCbCr color space is a pointwith no hue or saturation and if it were large enough to see it wouldappear as a gray spot with an intensity value of Y, which in this caseis a value of 128.

FIG. 22 shows 2D hue/color saturation histogram for a halictid bee. Theimage of the bee from which the data is derived appears in the upperportion of this figure. Note that the peak in the center of the Cb/Crhistogram corresponds to very low color saturation, which in this case,are the black stripes on the bee's abdomen and the darker areas alongthe edge of the thorax and head. The ridge or peaks radiating out fromthe central peak (heading left from the center of the surface), which isparallel to the Cb axis and is approximately between the Cr values of 15and 22, represents the yellow stripes of the abdomen and what is visibleof the yellow legs. The metallic green color of the head and thorax isrepresented by the scattered smaller peaks that lie in the region thatis less than 20 Cb and less than 15 Cr (upper left hand quarter of thematrix). The further the region is from the center of the space, themore saturated the color. For example, the ridge representing yellowindicates that a portion of it near the central peak is a very lightyellow (lots of white, nearly white) while the area near the left edgeof the histogram represents the brighter yellow colors. The bee fromwhich this image was generated was also used for one of ourdemonstrations and also is the left-most bee in the dorsal trainingimage of FIG. 26.

FIG. 23 shows values of the circular fit/compactness feature for threeclasses of geometric shapes. Note that the metric decreases as the shapebecomes less circular (down the columns) or less compact by elongatingor stretching the shape (across the rows).

FIG. 24 shows a flowchart of the general description of a method ofoperation 2400 of some embodiments the system. At block 2410, someembodiments of the invention include generating a background image of adetection surface (e.g., by obtaining an initial or earlier image of theactual collection surface, or by generating a synthetic image based on aspecification or assumption of what the background image should be, forexample, when using a standardized, pre-printed background). At block2412, some embodiments of the invention include placing insects or otherarthropods (or other objects to be identified) on the collection surface(e.g., by using a sticky surface and attracting the arthropods to thesurface where they land and become stuck, or by using a net or othercollection mechanism to catch the arthropods, then immobilizing thearthropods and placing them on a scanner surface). At block 2414, someembodiments of the invention include acquiring one or more images. If atbranch block 2416, it is desired to perform a training operation,control is passed to block 2420; else control is passed to block 2430.At block 2420, some embodiments of the invention include generatingcharacteristic or identifying features and/or silhouettes of the objects(for example, in some embodiments, these objects include arthropods thathave been pre-identified or classified by an expert entomologist), andcontrol is passed to block 2470. At block 2470, some embodiments of theinvention include saving the data regarding the pre-identified objectsinto a feature file and optionally into a silhouette file. If at branchblock 2416, it was desired to perform an identification-of-unknown(s)operation, control was passed to block 2430. At block 2430, someembodiments of the invention include analyzing the unknown image andpassing control to block 2440. At block 2440, some embodiments of theinvention include detecting an object (e.g., the unknown arthropod to beidentified) and passing control to block 2450. At block 2450, someembodiments of the invention include extracting features and optionallythe silhouette of the unknown object (e.g., arthropod) and passingcontrol to block 2460. At block 2460, some embodiments of the inventioninclude classifying the unknown arthropod by comparing its features withreference data from the feature file generated in an earlier trainingoperation (block 2420) and passing control to block 2480. At block 2460,some embodiments of the invention also include saving information as tothe unknown (e.g., its place and time of collection, the silhouetteand/or color-reference pixels, the classification that was determined,etc.) into a classified-unknown-arthropods section of a results file andthen passing control to block 2480. At block 2480, the arthropodclassification or identification have been made, and some embodiments ofthe invention include outputting or transmitting a report (e.g., togovernmental or commercial organizations, or to the user who requestedthe identification service).

FIG. 25 and FIG. 26 show two images used for generating the identifyingreference features. ScanDorsalTraining.bmp (FIG. 25) has the dorsal viewof eleven insects while ScanVentralTraining.bmp (FIG. 26) has theventral view of the same eleven individuals. The top row has two fliesof a syrphid species with yellow longitudinal stripes on its thorax; thesecond row has two asparagus beetles and a second species of syrphidfly; the third row has three halictid bees; the fourth row contains ablow fly; and the bottom row has two multicolored Asiatic Ladybirdbeetles.

FIG. 27A and FIG. 27B show two test images of the same ten insectindividuals. ScanDorsalTest.bmp (FIG. 27A) contains the test insectswith their dorsal side exposed to the scanner whileScanVentralTest.bmp—(FIG. 27B) has the ventral view of the insects. Thetop row includes two syrphid flies of the species with a yellow strippedthorax. The second row has a halictid bee and a second species ofsyrphid fly (right). The third row contains a blow fly (left) and ahalictid bee (right). The fourth row has two multicolored Asiaticladybird beetles. Bottom row includes two asparagus beetles.

FIG. 28A and FIG. 28B show the test case containing dorsal views of tengarden insects (FIG. 28A, as in FIG. 27A) and the successful detectionand recognition of these insects (FIG. 28B). A portion of the abdomen ofthe top left most insect, a syrphid fly, was detected as a separateobject, as was a portion of the right wing of the syrphid fly colored inblue. These two detections were rejected during connected componentsanalysis as too small. Objects were rejected if they were less than halfthe area of our smallest reference specimen, the asparagus beetle.

FIG. 29A and FIG. 29B show the test case containing ventral views of tengarden insects (FIG. 29A, as in FIG. 27B) and the successful detectionand recognition of these insects (FIG. 29B).

FIG. 30A and FIG. 30B show a test image of insects in clutter (FIG. 30A)and the output results image with the correct detection andidentification of the objects (FIG. 30B). The plant material has beenautomatically labeled red to indicate it belongs to the class of objectsthat are not of interest, and which is called OTHER. Note that thesyrphid fly at the top of the image is missing its abdomen and theasparagus beetle at the bottom of the image has lost its head andthorax.

FIG. 31 shows an image that simulates a snapshot from a previoussampling period. It will be used as a background image to compare with amore recently collected sample image. The image contains an asparagusbeetle (top), a multicolored Asiatic ladybird beetle (middle) and agreen ash seed (bottom).

FIG. 32A-32F show prototype silhouettes for garden insects. FIG. 32Ashows the syrphid fly species with a striped thorax, FIG. 32B shows anasparagus beetle and FIG. 32C shows a second species of syrphid fly withno stripes on the thorax. FIG. 32D shows a halictid bee, FIG. 32E showsa blow fly and FIG. 32F shows a multicolored Asiatic ladybird beetle.

FIG. 33 shows a test image of insects being overlapped by other insectsor clutter called Occ2A.bmp. Two asparagus beetles are abutting oneanother (top) while two multicolored Asiatic ladybird beetles touch oneanother in the middle of the image. Approximately half of a halictid beeis occluded by an ash seed (bottom).

FIG. 34—Successful detection and classification in the case of occlusionwhen the object doing the occluding can be subtracted from the currentimage by taking the difference between the current image and a previousimage that contains the occluding object. The second asparagus beetle(top left), the second ladybug (lower middle) and the halictid bee(bottom) were detected and correctly identified by the nearest neighborclassifier. The identification of the halictid bee was also confirmed bythe silhouette matching routine.

FIGS. 35A-35C show silhouette matching. The occluded halictid bee'ssilhouette matched best with a prototype silhouette of a halictid bee asshown in FIG. 35C. The silhouette of the known is colored blue while theunknown's silhouette is red. Where they overlap the pixels should appearpurplish. The image of FIG. 35A the left is the silhouette of theoccluded bee, while the silhouette of the halictid bee prototype ispresented in FIG. 35B.

FIG. 36 shows color coding of the best matches for three cases ofocclusion when there was no background image with information aboutpreviously collected insects. In this case the BugClassify programestimated the background image. The color coding indicates thataccording to the nearest neighbor classifier the pair of asparagusbeetles (at the top) best matched a blow fly while the pair of ladybugbeetles (in the middle) and the ash seed with a halictid bee (at thebottom) matched the syrphid fly species with the striped thorax.However, the matching metric of the nearest neighbor classifierindicated that none of these matches were good matches.

FIGS. 37A-37F show silhouette matching results for three cases ofocclusion. Each row represents the results of a different pair ofoccluded objects: two asparagus beetles (FIGS. 37A-37B on the top row),two ladybird beetles (FIGS. 37C-37D on the middle row), and a halictidbee occluded by an ash seed (FIGS. 37E-37F on the bottom row). In thecase of the beetles, silhouette matching detected and identified eachone. The best match among each pair of beetles is shown by the image onthe left while the second beetle was the next best match (image on theright). For the case of the occluded halictid bee (FIGS. 37E-37F on thebottom row) the prototype silhouette and representations of theadditional sample pixels for color are displayed on the lower left image(FIG. 37E, which shows a prototype silhouette (shown in blue) andspotprints (the green and black crosses and the yellow sideways Ts) fora halictid bee). These reference sample colors indicate green on thehead and thorax and the yellow and black stripes of the abdomen. Thebest overall silhouette match (including the color match) for the bee isshown in the lower right image (FIG. 37F).

FIGS. 38A-38C show three of the best silhouette matches for the occludedbee. In all three cases the prototype silhouette was a halictid bee. Thespurious correlations of the images FIG. 38C on the right and FIG. 38Aon the left were actually slightly better than the correct match (FIG.38B, the middle image) in terms of the number of silhouette pixels thatwere matched. However, the correct match was better overall becausethree of the six color pixels matched the original image's color, whilethose of the other two correlations did not match the color of the seedand were rejected.

FIG. 39 shows equipment setup used in some embodiments for testing theconcept of automated detection and identification of arthropods using adigital color camera as part of the system.

FIG. 40 is an image of a detection surface before weevils are“collected” or placed for identification on it.

FIG. 41 is an image of seven boll weevils used for training theclassifier. From left to right: weevil on its side, weevil on its side,weevil partially on its side and back, weevil on its back, weevil on itsside, weevil sitting on its posterior and a weevil on its abdomen.

FIG. 42 is an image of a detection surface after three weevils were“collected” by a detection device that is based on the described systemand software. This was the first of two test images. It is calledwst0.bmp.

FIG. 43 is an image of a second test image, wst1.bmp. Detection areaafter three additional insects, two more boll weevils and a cantharidbeetle, were “collected.” In this picture each insect is identified by alabel to its left, BW for boll weevil and CB for cantharid beetle.

FIG. 44 is an image output following processing. Three weevils weredetected, classified and counted. Detected regions that were classifiedas a boll weevil are colored green. Had there been any insectsclassified as OTHER than boll weevils, they would have been colored redby the software. See FIG. 45 for an example of how an insect other thana boll weevil was color coded in this experiment.

FIG. 45 shows an image output following processing. Five boll weevilsand one non-boll weevil were detected. Detected regions that wereclassified as a boll weevil are colored green and detected regions thatwere classified as non-boll weevil are colored red by the software.

FIG. 46 shows a distribution of the reference boll weevils (trainingweevils), unknown or test weevils and the cantharid beetle in a threedimensional feature space where the dimensions or features are totalarea (z-axis) and the two parameters that characterize the insectscolor, Cr (red saturation, y-axis) and Cb (blue saturation, x-axis).Note that each of the unknown boll weevils is relatively close infeature space to a reference specimen of a boll weevil, while thecantharid beetle is a significant distance away from the referenceweevils. For total area the cantharid is 9.2 standard deviations awayfrom the average area of the reference weevils, 3.1 standard deviationsaway from the average weevil's Cr value and 4.2 standard deviations fromthe average Cb value of the reference weevils. Based on Grubbs' test foroutliers using the area feature, the cantharid can be rejected asbelonging to the boll weevil population with a probability of error thatis less than 1%. Based on the Cb feature the Grubbs' test rejects thecantharid as a boll weevil with a probability of error that is less than2.5%. The Grubbs' test indicates that if the cantharid is rejected as aweevil based on its Cr value only, there is a probability of error alittle over 10%.

In some embodiments, a suite of image-processing and pattern-recognitionalgorithms implemented in software that enable the detection andclassification of arthropods with minimal human involvement and providerobust results under varying and complex conditions such as: arthropodsamong extraneous objects, arthropods in varying positions andorientations, overlapping specimens or occlusion, incomplete or damagedspecimens, and image artifacts such as shadows and glare. In someembodiments, both the detection and classification process takeadvantage of color information, namely hue and color saturation, inaddition to luminance or the intensity of reflected light. In someembodiments, classification includes two levels of processing: 1) aninitial statistical-feature-matching classifier for quick results on itsown or to act as a screening function to pass on more complexclassification problems to a second level classifier; and 2) acomputationally more complex syntactic classifier that deals withdifficult problems including clutter, incomplete specimens andocclusion. The statistical-based classifier extracts a hue andsaturation histogram, measurable size, shape, luminance and/or othercolor features and compares them to the same types of features similarlyextracted from reference specimens. This classifier provides a quick andefficient means of arthropod classification and is able to assign aconfidence level or metric to its classification decisions. In someembodiments, if the confidence measure of the statistical classifier isdeemed low compared to user defined thresholds, the statisticalclassifier passes the final decision to the second level classifier. Thesecond level classifier searches for structural details of thearthropod, normally the arthropod's silhouette, and spatially relatesthese structures to the location of patterns of color on the arthropod.

The extensive use of color information for both detection andclassification, luminance for classification and the two tierclassification approach enable a practical systems for the automaticdetection and classification of insects in the field or in laboratorysettings where there is little control over what is in the camera'sfield of view and how objects are arranged in that field of view.

The present invention integrates highly automated systems, which includedevices, processes, software and graphics, to acquire, process anddisplay images for the detection and classification of arthropods.Moreover, the present invention is able to detect and classify insectsand other arthropods under conditions that make counting and classifyingdifficult such as: 1) the presence of objects that could be mistaken forarthropods, which can be referred to as clutter; 2) the presence ofartifacts such as shadows; 3) the presence of overlapping or occludingobjects; and 4) incomplete insects due to injury or damage.

In some embodiments, the present invention's image-processing systemperforms both the automatic detection and classification of unknownarthropods. It performs despite the presence of clutter, occlusions,shadows, and the arthropods' appearing in a variety of positions andorientations. Thus, the present invention doesn't require a highlycontrolled environment where the only objects that can be detected arearthropods, and it will classify insects/arthropods regardless of theirposition and orientation. This means it can be used for a wide range offield applications. The present invention also uses color information ininnovative ways in both the detection and classification processes. Theaspects of color used in some embodiments are both hue (the dominantcolor or wavelengths) and color saturation (the purity of the color). Inaddition to color, some embodiments use luminance or relative lightintensity.

In some embodiments, the present invention includes a classifier thatdiffers from those of the other applications in a key way: it uses a twostage approach to detection and classification. The method for the firstlevel of classification is a classifier that uses statistical features(feature classifier). This classifier can be used alone for applicationswhere the user knows in advance that arthropod classification will berelatively easy (no clutter, no overlapping arthropods, and each speciesis very distinct in appearance) or if the user is concerned more withprocessing speed than accuracy. Otherwise, the statistical-feature-basedclassifier acts to screen out unlikely classes for an unknown so thesecond more computationally demanding classifier will not take as muchtime to generate an answer. The second-level classifier is referred toas a syntactic pattern or structural pattern recognition method.

In some embodiments, the system includes a database for the first levelclassifier. The database includes numerical data for classification thatare taken from known reference specimens of arthropods. The referencespecimens reflect the diversity in form and appearance of differentpopulations and species as well as represent the different positions andorientations that specimens can take. When the system is used toclassify unknown specimens the first level classifier comparesquantitative measurements of size, shape, luminance and color featuresfrom each unknown with those from the reference specimens (database).This classifier allows the user to use all of the available features orselect a subset of them based on the advice of human taxonomic experts.An experienced human taxonomist is more likely to develop a morereliable and robust approach to classifying arthropods than a computerprogram. The human insect taxonomists have a better way to assigntaxonomic importance to various features and can readily adjust theclassifier to very specific situations which an artificial intelligenceapproach is unlikely to do.

In some embodiments, a second level classifier does syntactic orstructural pattern recognition. Statistical-feature-based classifiersidentify objects in a strictly quantitative way and tend to overlook theinterrelationships among the components of a pattern (Tou, J. T., and R.C. Gonzalez. 1974. Pattern Recognition Principles. Addison-WesleyPublishing. pp. 377). Syntactic classifiers look to see if the patternof an unknown matches a known case by having the same essentialcomponents or features and that these components are linked or arrangedin an identical way. Some embodiments do this by using an innovativeapproach to extending a “silhouette matching” method by including colorinformation. In silhouette matching, the classifier translates, rotatesand scales the 2D silhouette of a known object over the silhouette ofthe unknown until it finds the best match. It repeats this process foreach possible known and then assigns the unknown to the class of theprototype or reference silhouette that had the best overall match. Insome embodiments, this method is extended by including sample points inaddition to those along the edge or silhouette of the object. Theadditional sample points are in fixed positions relative to thesilhouette. Each point inside the silhouette contains color informationfor that pixel or pixel neighborhood. The sample points are chosen tocapture any distinctive color patterns that an arthropod may have. Inthis way the present invention examines the interrelationship betweenthe general shape of the insect and its color by checking whether theunknown has the right colors in specific places relative to thesilhouette. The best match should not only have many of the pixels fromboth the known and unknown silhouette overlap or be a short distanceaway, but the colors of the internal reference points must also closelymatch. The extended silhouette matching classifier is superior to thestatistical classifier in that it can often find a correct match whenonly a portion of the arthropod can be seen while a statistical-featureclassifier will normally fail to fine the correct class under suchconditions.

Because the present invention is practical, robust, accurate and timesaving, it reduces the cost of sampling arthropod populations and freeszoologists, ecologists and pest-management professionals to work on moreproductive tasks. In addition, the present invention technology has verybroad application to object detection in general.

Conceptual Description of Process, Devices and Algorithms for AutomatedDetection and Classification of Arthropods

First, a general description of the present invention is provided. Thisincludes the general system configuration, general description of theoperation of the system, its devices and algorithms. Then the actualapplied use of the present invention is demonstrated for threeembodiments in detecting and classifying arthropods. Demonstrationssimulate the detection and classification of insects in situationssimilar to the following: 1) insects placed on a surface for automaticclassification in an ecology laboratory where students need to classifyand count insects; 2) at customs facilities, ports of entry or any otherareas where introduction of certain arthropods are being prevented; 3)where a pest management scout has emptied the contents of his collectionnet/tool on a surface for automatic classification and counting; and 4)insects stuck to a sticky, colored surface of a trap that may or may notbe baited with an attractant, such as a pheromone, and which is used tomonitor insects in the field.

General System Configurations:

The present invention's automated arthropod classification system, atits core, includes a detection surface where the arthropods to beclassified are found, an imaging device (scanner, digital camera orvideo camera) to collect their images, and a computer with software tooperate the imaging device, process and analyze the images and topresent the results. One embodiment of a simple system would be ascanner 621 or digital camera 611 that communicates directly with theuser's computer 699 using a cable 631 such as a USB (Universal SerialBus) connector or wireless communications link 639, as shown in FIG. 6.The user's computer is also referred to as the host computer since ithosts the software to control the imaging device and process the imagesfor the classification of arthropods. Thus, from the user's computer,one can: 1) control the settings for the scanner or camera; 2) requestan image immediately from the imaging device or else schedule theautomatic periodic collection of images; 3) automatically process theimages to detect and classify the arthropods; and 4) examine thecollected images and review the results of the automated detection andclassification.

Alternatively, in some embodiments, the imaging device can transmitimages to the host computer and receive instructions from the hostcomputer via a dialup modem connection, internet connection or awireless communication device. Some embodiments that use a camera alsoinclude an illumination device to facilitate the acquisition of imagesof the surface (detection surface 624) where the arthropods to bedetected are found.

More complex embodiments will rely on multiple imaging devices connectedvia a network (wired and/or wireless) to the user's computer. Thescheduling of the sampling and the processing of the images would stillbe done from the user's host computer. An even more advanced embodimentof an arthropod-sampling system, in some embodiments, includes manyindependent arthropod-detection units sending back the images andprocessed results to the user's computer. Each arthropod-detection unitwould include a trap or detection surface, illumination device, camera,camera lenses and filters, processor and communication device. Each unitin addition to collecting an image would do the detection andclassification processing with its own processor or CPU (centralprocessing unit) and then send compressed images and the results ofprocessing to the user's computer/host computer for review. The usercould adjust settings for the individual cameras, schedule the samplingtimes and set the processing parameters for each of the units from thehost computer, as well as review the results from the units.

In some embodiments, the host computer includes the followingoff-the-shelf, commercially available software:

-   -   1. Operating system.    -   2. Software to initiate image collection via an imaging device.    -   3. Microsoft Paint—examine input, intermediate and final result        images.    -   4. Microsoft Notebook—examine output text file and edit input        text file for some embodiment, sometimes called the feature        file.    -   5. Starbase's CodeWright—another text editor, to develop the C        code for applications of some embodiments and to examine output        text files and examine and edit the feature files.    -   6. Microsoft Visual C++—to develop and compile some embodiments'        executable programs, such as BugClassify.exe.

In some embodiments, the present invention software includes:

-   -   1. BugClassify.exe—executable software used to train a system's        statistical classifier and to process images for the detection        and classification of arthropods.

2. MakeSilh.exe—executable software integrated with a main programBugClassify.exe—executable software that takes a segmented image orlabeled image of the detected arthropods, and generates an imagecontaining the silhouette of each detected object (also called a“detection”). This is used for research and development usingsilhouettes for arthropod classification.

-   -   3. GetSilhCode.exe—executable software that extracts a        compressed representation of an object's silhouette, called the        chain code, and inserts the chain code into a special silhouette        file. This is used to develop prototype silhouette files and to        do studies with silhouette files.    -   4. TransSilh.exe—executable software that does silhouette        matching in place of BugClassify.exe when doing        silhouette-matching studies.

In some embodiments, the imaging device (scanner, digital camera)includes software to adjust camera or scanner settings such asbrightness, spatial resolution (dots per inch) and color resolution(number of bits of color) as well as request the collection of an image.

Specific configurations of some embodiments for the demonstrationsdescribed here appear in their respective sections.

General Description of the Operation of the System

This section describes an embodiment to configure a system to detect andclassify insects that are on a detection surface 624. The insects mayhave been collected by sampling insects from a habitat, for example byusing a sweeping net or other sampling device. The person places theinsects on the detection surface 624 of the system and has the insectsautomatically classified and counted. The described embodiment alsoworks when the insects that are to be detected and counted were trappedafter they flew or crawled onto a sticky detection surface 624 in thefield.

One embodiment of a system follows and is summarized in a flowchart 2400(see FIG. 24):

-   -   A. Generation of a background image.    -   B. Generation of arthropod-identifying reference features and        prototype silhouettes.    -   C. Acquisition of images of the unknown arthropods to be        detected.    -   D. Arthropod detection.    -   E. Feature extraction.    -   F. Classification of arthropods.

A. Generation of a Background Image

Normally, the first function is to collect a background image of thedetection surface 624 prior to placing insects on it. The scanner orcamera acquires an image of this surface without any objects on it. Thisreference image aids in the detection of the arthropods and otherobjects that will eventually appear on the surface. The background isused to look for changes in the quantity, hue and color saturation ofthe reflected light due to the presence of the arthropods. FIG. 40 fromExperiment 3 shows an example of a background image.

Although it is advantageous to include a background image or previousimage as input for accurate insect detection, a system doesn't requireit. As an alternative, software has the option of estimating what thebackground or detection surface 624 would look like in terms of colorand luminance in the absence of any arthropods. The terminology usedhere follows the RGB color model (Weeks, A. R. 1996. Fundamentals ofElectronic Image Processing. Wiley-IEEE Press. pp. 576). Software cancalculate the median R (red), G (green), B (blue) and gray-level valuesas well as their standard deviations from among all the pixels, and thenuse these values as estimates of the background detection surface 624.This is valid at least as long as most of the detection surface's areais visible and the one or more various background areas of detectionsurface 624 itself are each relatively uniform in color and luminosity(which each is, by design, in some embodiments). An estimated backgroundimage is created by using the original RGB values of each pixel,provided that they are within a specified number or fraction of astandard deviation of the median RGB values. Otherwise, if any of thepixel's RGB values differ significantly from their median values, it maybe part of an object, and its RGB values are replaced with the medianvalues. Alternatively, all the pixels of the estimated background imagecan use the median RGB values.

After acquiring the background or reference image, there may be a needto enhance the image before any further processing can be done. Thiscould also be true for the images containing the insects to be countedand classified. Image enhancement to correct for distortions or noise isan optional feature in some embodiments.

If the angle formed by the camera's line of sight and the detectionsurface or insect(s) deviates from the perpendicular, there can besignificant distortion due to perspective. It is possible that in someapplications, particularly where a person may wish to examine theimages, perspective distortion is unacceptable. In this case, thepresent invention can map each pixel's value via geometrictransformation to a different coordinate system (different row andcolumn) that corresponds to a top-down or perpendicular view. Duringcalibration, the present invention measures the real-world coordinatesof four points in the distorted image space. With these four points thepresent invention can solve for the coefficients that will permitsoftware to map pixels in the image into a normal view (Russ, J. C.1995. The image processing handbook. 2nd edition. CRC Press. pp. 674).Gaps or missing values in the transformed image can be filled in throughinterpolation.

Image enhancement may also be needed if there is a significant amount ofnoise in the image. Two basic approaches can be used to filter outnoise: temporal or spatial smoothing. When a sequence of images can becollected over a short period of time, an enhanced image can be createdby replacing the value at each pixel location with either the arithmeticaverage or median of that pixel from among the replicated images. Theresulting smoothed image will be enhanced and have little noise,provided that the scene does not change between images and that thenoise is nearly random over time at each pixel. Using the arithmeticaverage for each pixel is desirable in most cases, as it requires fewercomputations than calculating the median.

When it is not practical to collect several images to reduce noise, aspatial filter can be applied. In this case the pixel of concern isreplaced with the arithmetic average of its value and those of itsneighbors. Alternatively, the pixel can be replaced with the median ofthe values in its neighborhood. While the arithmetic average iscomputationally quicker than calculating the median, the median issometimes desirable as it is less likely to blur the image alongcontrasting areas or boundaries within the image.

Although the above noise-filtering techniques are standardimage-processing techniques, some embodiments have two modifications tothese approaches to noise reduction. To preserve as much of the originalimage information as possible in the case of spatial averaging, someembodiments use the original pixel value except when the differencebetween the averaged value and original is large. When there seems to bea significant difference, some embodiments use the averaged or medianvalue. In this way, the original information is retained except forcases where noise may have caused a questionable value. In someembodiments, a second approach deals with the averaging of color. Eventhough the previously described filtering methods work well forblack-and-white images or the luminance portion of a color image, insome image formats or color models just averaging each color componentcan lead to unintended or distorted color. For example in the RGBformat, if one takes the average of a reddish colored pixel (R=248, G=8,B=112) and greenish pixel (R=8, G=248, B=112) it would result in a graypixel (R=128, G=128, B=112). This may not be what was intended. The twoinput pixels had high color saturation and the resulting average has avery low saturation. One way color distortion can be avoided whensmoothing the image values is to use the color components of the pixelwith the median luminance value.

B. Generation of Identifying Reference Features and Prototype Silhouette

Before some embodiments can identify unknown arthropods, there must be aset of features and silhouettes (optional) from known or identifiedarthropods which can be compared with the features and silhouettes ofthe unknown arthropods. This section describes the features andsilhouettes. At the end of this section is a list and description of thecommands and functions that some embodiments use to generate thereference features and prototype silhouettes.

Some embodiments use a collection of statistical Identifying ReferenceFeatures extracted in advance from known arthropods or referencespecimens. A set of each of these Identifying Reference Features istaken from each reference specimen and they are used by the system'sstatistical classifier to identify the unknown arthropods. A pluralityof such feature sets are collected, in some embodiments, from specimensof each arthropod species that the system is expected to encounter orrequired to recognize. This insures that the system includesrepresentatives of the natural variation among individuals of a speciesand the different orientations in which the arthropods may appear. Insome embodiments, the sets of reference features are stored in thecomputer's memory as a file which is called a feature file.

In some embodiments, features are extracted from images of identifiedarthropods by the same processes (functions C, D and E) as describedbelow, for the classification of unknown arthropods. These featuresbecome part of a database of known arthropods. In some embodiments,features characterize each reference arthropod and fall into one of fourtypes of information:

-   -   1. Size: This set of features includes: a) total area; b)        perimeter; c) the length of the major axis (body length); d) the        length of the minor axis (body width); and e) the minimum        rectangular area that bounds the detected or labeled area.    -   2. Shape: This set of features includes: a) the ratio of the        total area to the minimum bounding rectangular area (measure of        how rectangular the object is); b) 4(pi) times the total area        divided by the perimeter square (a measure of how circular and        compact the object is); and c) height to width ratio or major        axis to minor axis ratio (a measure of elongation).    -   3. Luminance: Information on the brightness or relative        intensity of the reflected light from the arthropod is currently        described by two features: a) the average luminance or        gray-level value of each pixel of the detection relative to its        background; and b) the coefficient of variability in the        relative gray-level values among the detection's pixels. The        coefficient of variability is the statistical standard deviation        divided by the mean and expressed as a percentage. Some        embodiments use a histogram of the luminance values of the        arthropod's pixels as a way of characterizing its reflectivity.    -   4. Color: In some embodiments, a Color Feature or 2D hue/color        saturation histogram provides a simple and practical way to        summarize the colors of arthropods. This Color Feature invention        is an improvement for object identification and is derived from        the standard color format called the YCbCr color model        (Weeks 1996) which is used in video and television.

In some embodiments, the color feature (feature vector) is used toprovide a simple, powerful and practical way to summarize the color ofan arthropod or insect that is independent of rotation in the image aswell as scale. The color information associated with each pixel of thearthropod is translated in to YCbCr color space where Y represents thepixel's luminance or brightness and Cr and Cb represent the colorsaturation level for red and blue, respectively (Weeks 1996). The hue,for a point in the CbCr space, is represented by the angle formed by thex-axis and a line from that point to the center of the space. Thedistance from the point to the center of the color space represents thelevel of saturation for that point. The center of the space has no colorand represents the gray-scale value or luminance of the pixel.

FIG. 21 illustrates an example of a portion of a YCbCr color space whereY is kept at a constant gray-level value. In some embodiments, usingthis concept of the YCbCr color space, the software routine,BugClassify.exe, is programmed to generate a 2D histogram where the rowsrepresent the Cr value and the columns the value of Cb. Each bin of thehistogram contains the percentage of the pixels associated with thearthropod that have that combination of Cr and Cb or that combination ofhue and color saturation. FIG. 22 graphically shows an example of the 2Dhue/saturation color histogram generated from an image of one colorfulinsect (halictid bee). This figure also illustrates how the 2D histogramrepresents quantitatively and qualitatively the metallic green, yellowand black color of the halictid bee.

In some embodiments, the Cr and Cb values of each image pixel arerepresented by an eight-bit value. Therefore, it would be natural tohave a histogram with 256 columns and 256 rows. However, most of thebins will be empty (contain a zero, as there were no pixels with thatcombination of Cr and Cb). To avoid a sparse matrix, that is a histogramwith most of the elements having a value of zero, some embodimentsgenerally use only the upper 5 bits of each Cr and Cb value. This savesmemory space and reduces the time or number of calculations needed toclassify insects when a comparison is made of the matrix of each unknowninsect with the matrices of the reference insects. Thus, the size of thematrix can be altered to allow for varying color resolutions.

The size and shape of the CbCr color space changes with luminance.Therefore, the lighting conditions for collecting images of thereference insects and the unknowns (i.e., insects to be detected andclassified) are kept as nearly identical as possible to insure a validmatch. In some embodiments, several other color space models are usedfor a 2D hue/saturation color histogram, such as the HIS, YIQ, HLS, HSV,CMY or L*u*v* color models. The L*u*v* color model would be a goodcandidate if illumination complicates some embodiment's color-matchingapproach. Ong et al. (Ong, S. H., N. C. Yeo, K. H. Lee, Y. V. Venkatesh,and D. M. Cao. 2002. “Segmentation of color images using a two-stageself-organizing network.” Image and Vision Computing 20(4), pp. 279-289)used the L*u*v* color space to determine the dominant colors in imagesfor segmentation (labeling the pixels making up an object). They foundby using the L*u*v* color space, the influence of illumination on colorswas greatly reduced.

Color histograms have been used in the past for segmentation and forcharacterizing images so that an image can be matched quickly withanother in an image database. For example, Chai et al. (Chai, D., and A.Bouzerdoum. 2000. “A Bayesian approach to skin color classification inYCbCr color space.” IEEE Region Ten Conference, (TENCON'2000), KualaLumpur, Malaysia, vol. II, pp. 421-424, September 2000) used a 2D colorhistogram based on YCbCr color space to segment human faces in images.They used the 2D color histogram to create a conditional probabilitydensity function for skin color which was then used to decide whetherindividual pixels in an image belonged to human skin. The inventorsbelieve the present invention's 2D hue/saturation color histogram is thefirst application of a 2D color histogram as a feature for theidentification of objects within an image. A 1D color histogram was usedby Di Ruberto et al. (Di Ruberto, C., A. Dempster, S. Khan, and B.Jarra. 2002. “Analysis of infected blood cell images using morphologicaloperators.” Image and Vision Computing 20(2), pp. 133-146) as a featureto distinguish red blood cells from other blood cells. They mapped thepixels of reference red blood cells to a 1D color histogram where theelements of the histogram represented 256 colors taken from an HSV colorspace. This reference histogram then was matched with 1D colorhistograms extracted from unidentified blood cells. A 2D color histogramshould be more effective at distinguishing subtle color differencesamong different insect species than just a 1D color histogram.

Some embodiments have the option of applying a second level ofclassification in addition to the statistical-feature classifier. Someembodiments optionally generate a second computer file called the“prototype silhouette file” for the syntactic-silhouette-matchingclassifier. The silhouette file contains numerical information thatencodes the 2D silhouette pattern along with color reference points foreach of the known reference specimens. The encoded form of thesilhouette is referred to as a chain code (Ballard, D. H., and C. M.Brown. 1982. Computer Vision. Prentice-Hall, Inc. pp. 523). Chain codesaves a great deal of computer memory compared to listing the x and yposition of each pixel making up the silhouette of the arthropodreference. A graphic example of the content of one of these prototypesilhouette files is given in FIG. 32 and an example prototype silhouetteillustrating the color reference points in addition to the silhouette isgiven in FIG. 37.

Prior to extracting the reference features and/or the prototypesilhouettes, some embodiments are configured in one of two ways:

-   -   Configuration 1    -   A. Imaging device: scanner.    -   B. Detection surface: scanning surface of scanner.    -   C. Computer:        -   1. DOS operating system.        -   2. Software to operate scanner.        -   3. Microsoft Paint—used in some embodiments to examine            input, intermediate and final result images.        -   4. Microsoft Notebook—used in some embodiments to examine            output text file and edit the input text file, called the            feature file.        -   5. Starbase's CodeWright—to develop the C code for            applications and to examine output text files and examine            and edit the feature files.        -   6. BugClassify.exe—executable software used in some            embodiments to train the system's statistical classifiers            and to process images for the detection and classification            of arthropods.        -   7. MakeSilh.exe—executable software integrated with the main            program BugClassify.exe of some embodiments. This software            takes a segmented image or labeled image of the detected            arthropods and generates an image containing the silhouette            of each detected object. This is used for research and            development with silhouettes for arthropod classification.        -   8. GetSilhCode.exe—executable software that extracts a            compressed representation of an object's silhouette called            the chain code and inserts the chain code into a special            silhouette file. Some embodiments use this to develop            prototype silhouette files and to do studies with silhouette            files.        -   9. TransSilh.exe—executable software that does silhouette            matching in place of BugClassify.exe when classification of            arthropods by silhouette matching is required.    -   Configuration 2    -   A. Imaging device: Digital camera.    -   B. Detection surface: Sticky surface where insects are trapped.    -   C. Computer:        -   1. DOS operating system.        -   2. Software to initiate image collection via an imaging            device.        -   3. Microsoft Paint—examine input, intermediate and final            result images.        -   4. Microsoft Notebook—See above.        -   5. Starbase's CodeWright—See above.        -   6. BugClassify.exe—See above.        -   7. MakeSilh.exe—See above.        -   8. GetSilhCode.exe—See above.        -   9. TransSilh.exe—See above.

In other embodiments, Windows, Linux, Unix, or other suitable operatingsystem may be used. Other image manipulation and text programs may alsobe used.

To operate the system, the user executes the following procedure toextract the features and create the feature file.

The user places classified reference specimens on the surface of theimaging device, acquires images of the specimens using the imagingdevice and saves these images as files in the computer's memory. Usingthe software that comes with the scanner or camera, the user clicks onthe capture image button. Once the image is captured the user pressesthe save function. The software requests a file name and image format.The user types in a file name of the user's choice for the image andthen must select the bitmap format, BMP, for the image file beforehitting the save button. The reference features are generated byexecuting the invention's detection and classification software, calledBugClassify.exe, in what is referred to as the “training mode,” with theimage of the known prototypes or reference insects as input. In the“training mode” the software executes in exactly the same manner as inthe “detection/classification mode” until the last function,classification. Instead of trying to classify the insects, in the“training mode” the software saves the feature set associated with eachknown insect to a file called the feature file (Identifying ReferenceFeature file).

To execute the program, BugClassify.exe, to generate the referencefeature file the user must bring up a DOS window in a Microsoft Window'soperating system. There are several ways to invoke BugClassify on thecommand line in order to create a feature file, four of which are shownhere:BugClassify input_reference_image_filenameinput_background_image_filename TrainingmodeorBugClassify input_reference_image_filenameinput_background_image_filename feature_filename—trainorBugClassify input_reference_image_filename estimatebackgroundtrainingmodeorBugClassify input_reference_image_filename output_background_estimatefilename trainingmode—background

On the command line, BugClassify must be typed in, followed by at leastthree arguments or character strings. Those arguments that are shown initalics indicate names that the user chooses. Those words that are notin italics are key words that BugClassify recognizes. Optional argumentsthat follow the first three always start with a dash to identify them asoptional parameters. The first argument is always the name of the imagefile, that is, either the image containing the reference arthropods fortraining or the unknown arthropods to be counted and classified. Thesecond file is always an image that shows what the background lookedlike before arthropods were placed on the surface or the image from thelast sampling period. If the user does not have a background image, sheor he can substitute a file name with the string, “estimatebackground”(third example above). This informs BugClassify that there is nobackground image and that it must estimate one from the input image; bydefault BugClassify will save this estimated background image to a filecalled BackgroundImg.bmp. Alternatively, if the user wants a differentname for the estimated background image, he or she types this filenameas this second argument, and anywhere after the third argument they mustenter the optional argument “-background” to indicate that a backgroundimage must be calculated (see fourth example above). The third argumentis the name of the feature file to be used during detection andclassification. Since a feature file is not present at the time oftraining, one needs to be created. This can be done in one of two ways,as shown in the first and second examples above. A feature file can begenerated by typing the string “trainingmode” as the third argument (seefirst, third and fourth examples above). This tells BugClassify to usethe input image as an image containing reference specimens. By defaultBugClassify will save the feature sets to a file called, TrainFile.txt.If the user wishes a different name for this feature file, she or heshould use the name that is wanted as the third argument, but must alsoadd the optional string, “-train”, anywhere after this third argument(as in second example above). The optional string, “-train”, tellsBugClassify to execute in the “trainingmode.”

Once the feature file is generated it must be edited by the user byadding the species name and species code number to each referenceinsect's set of features. This allows the classifier to assign thespecies name or classification associated with the feature set that bestmatches the features of the unknown. To edit the feature file, the useropens the file with any text-editing program, Microsoft Notebook forexample, and types two lines before each set of features (there is ablank line between each feature set). The first line must contain thename of the species as a character string. The second line is a numberto represent the species.

Additional functions are necessary to generate a prototype silhouettefile if the user plans on using the second-level classifier,syntactic-silhouette matching, in addition to the statistical-featureclassifier. First the user takes one of two intermediate output imagesto create a silhouette file of the reference specimens. These twointermediate images were generated when BugClassify made the referencefeature file. The intermediate images are: an image of the segmenteddetections (binary image where the background is black and the pixels ofthe detected objects are white) called, SegmentedImg.bmp, or a labeledimage of the detections (image where the background is black or zero invalue and the pixels of each detected object are assigned a positivevalue unique to each detection), called LabelImg.bmp. Either of thesetwo files is input for a command called MakeSilh.exe. The user types thefollowing command line in a DOS window to generate the silhouette image:MakeSilh LabelImg SilhouetteImageNameorMakeSilh SegmentedImg SilhouetteImageName

The second argument for MakeSilh.exe is the output silhouette file andit is shown in italics above to indicate that the user chooses the nameof this file. The next function takes the silhouette image of thereference specimens and generates a chain-code representation of each ofthe silhouettes. This is done by entering the following command line ina DOS shell:GetSilhCode SilhouetteImageName SilhouetteFilename.

The first argument to GetSilhCode is the name of the silhouette imagethat was generated by MakeSilh and the second argument is the name theuser chooses for the prototype silhouette-chain-code file. Thechain-code file is manually edited next. The species code number isadded as a line before each of the chain codes. Also the color referencepoints must be appended to each chain code. First the user appends thenumber of reference points followed by information on each of the colorreference pixels. For each pixel, its x and y coordinates are entered,its hue, and its saturation value. A space is typed between each value.In some embodiments, the reference points are selected and their x and yposition and their RGB values obtained by viewing the referencespecimens in the original raw image with Adobe's Photoshop Version 7.0.Photoshop provides the x and y value and the RGB values of each pixelpointed to by the cursor. The RGB values are manually converted to hueand saturation values by the equations given in Section D, Arthropoddetection (below).

When the software is executed to detect, classify and count arthropods,this reference feature file and the optional prototypesilhouette-chain-code file are included as input to the software alongwith the raw images.

C. Acquisition of Images of the Unknown Arthropods to be Detected

The scanner or camera acquires one or more images of the arthropods tobe detected and classified on the detection surface 624.

D. Arthropod Detection

This function involves labeling those pixels from the acquired arthropodimages that appear different from the corresponding pixels of thebackground image and are thus likely to belong to an arthropod orclutter. The system looks for differences in luminance, hue and colorsaturation between the corresponding pixels. Where a pixel is moderatelydarker than the background the pixel may represent a shadow and it isnot labeled as part of an object unless there are other indications thatit is an object and not a shadow, such as a change in hue. The labeledpixels are then connected into continuous regions or blobs by a standardimage-processing technique, connected-components analysis (such asdescribed by Ballard and Brown, 1982). As part of connected-componentanalysis, labeled regions that are too small in area are discarded. Thisfunction removes much of the false detections associated with noise andother artifacts.

The background image and the image with the unknown arthropods arecollected normally in a bitmap format where each pixel has 8-bit valuesfor the R, G and B color components. The RGB components are transformedto create separate intensity, hue and saturation images by firsttransforming them to the three components of the YCbCr color model. Theequations for these transformations are as follows:Y=0.299R+0.587G+0.114BCr=0.701R−0.587G+0.114BCb=−0.299R−0.587G+0.886Bwhere Y is the luminance or intensity of the pixel and Cr and Cb arecolor components of the YCbCr color model. Hue and saturation are thenderived from Cr and Cb by the following formulas:Saturation=square root (Cr²+Cb²)Hue=arc tan(Cr/Cb)

In some embodiments, hue is not defined when the saturation level iszero. Zero saturation means there is no color information and that thecolor appears as a grayscale value which can range from black to white.

Once intensity, hue and saturation images have been calculated for thecurrent and previous images, a difference image can be generated foreach image type. The previous image's luminosity values are subtractedfrom those of the current image to generate an absolute intensitydifference image. The same procedure is applied to the hue and colorsaturation images.

A threshold is applied to each of these three difference images.Differences greater than the threshold are labeled as significant andmay be part of an arthropod, while those pixels with values less than orequal to the threshold are labeled as background pixels. This process ofseparating the background pixels from the objects to be detected isreferred to as segmentation.

The threshold applied to each of the three difference image types is nota fixed value. It is adaptively calculated for each image type. Ahistogram is created for each difference image and from it a defaultthreshold is calculated by assigning the threshold to the differencevalue where only a small percentage of the pixels exceed this value.Setting the default threshold to the value where 15 percent of thepixels exceed the threshold generally works well. Next, an attempt ismade to improve upon the default threshold by searching the histogramfor a better threshold. An inflection point is sought where the value inthe histogram bin levels off or increases after having declined over theprevious bins. The first function is to start at the zero-differencebin, and search for the peak in frequency by examining the bins oflarger difference. From the peak difference, difference values inlarger-numbered bins are then searched for an inflection point or untilan empty bin is encountered. This inflection point becomes the thresholdunless it is considerably larger than the default threshold, in whichcase the threshold is assigned the default value.

A final detection or segmentation image is created by combining theresults of the three thresholded difference images. A logical-ORoperation is performed, of the three binary difference images, exceptwhere the intensity difference indicates the pixel could belong to ashadow, i.e., when the current pixel-intensity value is somewhat darkerthan the intensity of the background or previous image. When theintensity difference falls within the range of values that arecharacteristic of a shadow, the software labels the pixel as shadowedbackground unless the hue has significantly changed or the saturation ofcolor has significantly increased. In the latter case, the pixel isassigned to the detected object.

Arthropod detection need not be limited to just the processing of colorimagery. While color images offer more information for detecting andrecognizing arthropods, a black-and-white camera is cheaper and thus maybe preferable for situations where it is known the arthropods will beeasy to detect and classify. Detection for black-and-white imagery wouldbe the same as described above but the algorithms would be utilizingonly the luminosity or intensity image component and not the hue andsaturation images.

Following segmentation or the labeling of individual pixels, the labeledpixels must be grouped into regions, objects or blobs that correspond tothe arthropods. This can be done by the standard connected-componentsalgorithm. The algorithm scans pixels from the top row moving from leftto right across each pixel row until it encounters the pixel at theright-most column of the bottom row. The input to the algorithm is thebinary segmented image described in the previous paragraphs. The outputwill be grouped pixels where each grouped non-background region islabeled with its own unique non-zero identifying number while thebackground pixels are set to be zero. This output image is referred toas the labeled image.

As the algorithm scans through the segmented image it stops at eachnon-background pixel of the segmented image and assigns thecorresponding pixel of the output label image a non-zero number. If thepixel does not have a labeled neighbor directly above it or to its leftthe count of the number of labeled regions is incremented by one andthis value is assigned as the label number for this new region. If thesegmented pixel has a labeled neighbor above it, the algorithm assignsthe region label number of that neighbor as they are both members of thesame continuous region. If the neighbor above the segmented pixeldoesn't belong to a region but the neighbor to the left belongs to alabeled region, the pixel is assigned the label number of its leftneighbor, as they are connected to the same blob. If both the upper andleft neighbor have a label number but they are different, the pixel isgiven the upper pixel's label number, as it has precedence, and a recordis kept that these two labeled regions are connected and thusequivalent. During a second pass of the output image these twoequivalent regions will be merged.

After scanning through the segmentation image and assigning numbers toall the labeled regions a second pass is made through the output image.Wherever a non-zero label value is encountered that is equivalent to apreviously labeled region it is changed to the previous region's value.The count of the total number of labeled regions must also be adjustedby subtracting out the redundant or equivalent region.

While scanning the labeled image during the second pass a count is keptof the total number of pixels in each labeled region. Once this is doneany regions can be removed that are deemed to be too small by settingtheir pixel values in the labeled output image to zero and decrementingthe region count by one. The minimum pixel area for a labeled region canbe altered by the user, depending on the size of the arthropods ofinterest.

In some embodiments, the present invention implements the case of “fourconnectivity.” Four connectivity defines that a pixel is part of acommon region if any of the following four neighbors has also beenlabeled: the pixel above the pixel of concern (same column, precedingrow); the pixel below the pixel of concern (same column, next row); theleft neighbor (same row, preceding column); and the right neighbor (samerow, next column). It is also possible to execute connected-componentsanalysis with “eight connectivity.” In eight connectivity, in additionto lumping non-background pixels with their neighbor above, below, tothe right and to the left, the algorithm also looks at the neighboringpixels above and to the left and right, as well as pixels below and tothe left and right (the four diagonal neighbors). Eight connectivitytakes more computational time and may not yield significantly betterresults.

Within the labeled regions corresponding to the detected arthropodsthere may be holes that the process considered to belong to thebackground. Good examples would be the missing portions of the twodetected ladybird beetles which were caused by glare and are shown inFIG. 8 (image on right). These holes can be filled in by applyingconnected components an additional time. Prior to executingconnected-components analysis to label the non-background pixels,connected components can be applied to do just the opposite, label thebackground-pixel regions only. Any small background labeled regionsbelong to holes within the detected areas corresponding to thearthropods. These small background regions can then be used to fill inthe segmented image before the process calls the connected-componentsregion to label the detected arthropods. This process of filling holeswithin the detected regions was not done for the experiments describedin this document. This function was not incorporated into the softwareused in these experiments.

In some embodiments, arthropod detection is done by invoking thesoftware program called BugClassify.exe. From the computer, the usertypes the following command in a DOS shell:BugClassify input_image_filename input_image_background_filenameinput_feature_filename

If the user does not have a background or previous image and wants thesystem to estimate one, he or she needs to change the second argument tothe string, “estimatebackground”, or the name of the background estimateimage to be created plus the additional argument, “-background”, as wasdescribed in section B. The program by default discards detections thatcover less than 40 pixels in area. If the user wants to change thatvalue, he or she must add the optional argument anywhere after the thirdargument as follows, “-minsize” N, where N is area the user selects forthe minimum detection size.

E. Feature Extraction

The detection image containing the labeled regions created byconnected-components analysis is used to extract statistical features(size, shape, luminosity and color) and the silhouette (optional) foreach labeled region. Scanning over each labeled region the various sizeand shape features are counted and calculated and the silhouette'spattern is extracted into an encoded form or chain code. The sizefeatures that are calculated are: a) total area; b) perimeter; c) thelength of the major axis (body length); d) the length of the minor axis(body width); and e) the minimum rectangular area that bounds thelabeled area. The shape features include: a) the ratio of the total areato the minimum bounding rectangular area (measure of how rectangular theobject is); b) 4(pi) times the total area divided by the perimetersquare (a measure of how circular and compact the object is); and c)height to width ratio or major axis to minor axis ratio (a measure ofelongation). Features are also extracted relating to the quantity andquality of the light that is reflected from the arthropod's body. Therelative intensity-of-light or luminance features are: a) the averagedifference in luminance between the arthropod's pixels and thecorresponding pixels of the background image; and b) the coefficient ofvariability in the difference in luminance. In some embodiments, thequality of light or color reflected by the arthropod is captured by the2D hue/color saturation histogram which is considered as a feature orcompound feature (feature vector).

In some embodiments, the first shape feature listed above is referred toas the rectangular fit feature. It gives an idea of how rectangular inshape an arthropod is. This feature is calculated by dividing the totalarea of the object by the minimum sized rectangle that surrounds orencloses the object (referred to as the minimum bounding rectangle). Fora perfectly rectangular shape this ratio will be 1.0, and this feature'svalue will become smaller as the object becomes less like a rectangle inshape.

In some embodiments, the second shape feature listed above is called thecircular fit or compactness feature. It is also known as theisoperimetric quotient, which is defined as 4(pi) times the total areadivided by the square of the perimeter (Russ 1995). In some embodiments,this feature is used to measure how close to a circle and how compact anarthropod's shape is. This feature is at a maximum value, 1.0, for acircle, as both the numerator and denominator are equal to 4(pi)²r². Asan object's shape deviates from a circle the value of this featurebecomes smaller. Since a circle is the most compact shape, that is, ithas the smallest perimeter relative to its area for an enclosed object,this feature also measures compactness. Therefore a large feature valueindicates a compact object shape while a small value indicates that anobject is not compact, that is, it is flatter or thinner than a shapewith a larger value.

As can be ascertained from FIG. 23, if one approximates a circle withequal-sided polygons, the circular fit, or compactness, approaches thatof a circle as one adds more sides (triangle, square, hexagon (notshown), and octagon (not shown) have values of 0.604, 0.785, 0.842 and0.948, respectively). If one stretches a polygon, it becomes lesscompact and appears less like a circle, and the circular fit/compactnessmetric naturally decreases.

In some embodiments, to generate the Color Feature, for each labeledpixel of a region, the software extracts the color components Cr and Cb(which characterize its color hue and saturation, see Weeks, 1996). TheCr and Cb values from that pixel in the corresponding original inputimage are used in some embodiments to fill in the 2D hue/saturationcolor matrix that is created for each labeled region. In addition tocolor information some embodiments also utilize features that summarizethe luminance or gray-level values associated with the insect's image,such as the average gray level.

In addition to these features for classification, other statistics forother aspects of image processing are extracted from the labeledregions. These additional statistical measures include location featuressuch as the x,y extents of the object (x and y maximums and minimums) inthe image space, the x,y position of the object's center, called thecentroid, and the object's orientation (angle of major axis with respectto the x-axis). The minimum and maximum x and y coordinates describe therectangular region where the detection is located. The centroid is theaverage x and y value of the pixels that make up the detection's area.It tells the program and the user (normally the centroid is listed inthe text output) where the center of the detection is within the image.In some embodiments, an object's orientation refers to the angle thatthe major axis makes with respect to the x-axis. For example thearthropod's body may be facing up in the image, 90 degrees, or facingright parallel to the x-axis, 0 degrees.

Some embodiments provide a system that is flexible and can be customizedto specific situations where arthropods need to be classified. Althoughsome embodiments calculate many prototype features, (see previoussection B or first paragraph of this section), the user may choose touse only a few for specific classifications. How many and which featuresare chosen to be used depends on the application. Generally, morefeatures are used as more known arthropod species are added to aproprietary database of known arthropods. The program BugClassify bydefault uses four of the 11 features just described. They are the totalarea, the circular fitness feature, the average luminance and the ColorFeature. The user can select all or a subset of these features byincluding the optional argument, “-featsel”, on the command linefollowed by a list of the numerical codes for each feature. For example,in some embodiments, “-featsel 1,9,11” tells the program to use totalarea, the average luminance and the color feature in the statisticalclassifier. The numerical code for the features is the same as the orderin which the features were presented in the first paragraph of thissection.

F. Classification of Arthropods

The features extracted from each of the unknown arthropods (done infunction E) on the detection surface 624 are compared to each of thereference set of features generated by function B. In some embodiments,each unknown is classified by the statistical-feature classifier, whichis a modified version of the single nearest-neighbor algorithm (1NN)(Tou and Gonzalez, 1974). The unknown is assigned to the class belongingto the reference whose feature set is closest in the N dimensional spacedefined by the N features (best match). FIG. 46 shows an example of athree-dimensional feature space with the distribution of some referencespecimens and unknowns in that volume. Some embodiments of the 1NNclassifier differ from the standard version in the decision it makesonce all the distances to the various reference specimens have beenmade. Rather than just assign the class of the nearest reference infeature space like a typical 1NN classifier, some embodiments of theclassifier have options. The user can specify a threshold(s) whichdistinguishes good matches from poorer matches. If the distance infeature space to the best match is less than the threshold, then it is agood match and the classifier assigns that reference's class to theunknown. If the distance in feature space exceeds this threshold or ifthe difference in one of the key features is greater than the thresholdfor that feature, the classifier considers other alternatives. If thematch is poor, some embodiments reject the detection as not belonging toany class associated with the feature sets in the input feature file andassign the unknown to the class of unidentifiable objects called OTHER,or it can request further processing with the second-level,syntactic-silhouette-matching classifier.

BugClassify.exe decides whether the best match from the 1NN classifieris a good or poor match by doing the following. The user can choose athreshold that limits how different a feature may be between the unknownand the best matching reference specimen. The threshold is expressed asthe difference in the feature values divided by the value of thereference specimen. If any individual feature exceeds this threshold thematch is considered poor and the unknown is either assigned to the classOTHER or the decision is passed on to the syntactic-silhouette-matchingclassifier. The default threshold requires a difference of 1.0 orgreater (difference of 100% or more) to reject the 1NN classifier'sdecision. The user can alter this threshold with the optional argument,“-MaxFeatDist F”, where F is a floating point value of zero or greater.While each individual feature may not indicate a poor match there canstill be a poor match overall. Therefore, some embodiments include asecond threshold for the overall match in feature space. If the overallEuclidean distance exceeds a threshold value, the best match isconsidered poor. The default value is 0.5, which is equivalent to thefeatures' having an average difference of 50% or more. The user canchange this threshold by adding the following optional argument,“-AvgFeatThrs F”, where F is a floating point value that can be zero orgreater. Rather than set a limit on the quality of the nearest neighborclassifier's match as a percentage difference from the best matchingreference specimen, some embodiments replace these threshold metricswith actual confidence levels based on statistical tests. In someembodiments, for each feature a statistical test is conducted to see ifthe unknown is a statistical outlier and should not be considered as amember of the population of the best matching class. There are severalsuch statistical tests to choose from. Some embodiments use Grubbs' testfor detecting outliers. Grubbs' test calculates a ratio called Z, whereZ is equal to the difference between the unknown's feature value and themean value of the reference specimens of the class that best matches theunknown, divided by the standard deviation among the reference specimensof the best matching class. The mean and standard deviation has to alsoinclude the unknown in it. If Z exceeds a critical value for a givenconfidence level, some embodiments reject the decision of the 1NNclassifier. The user can choose among several confidence levels. Theuser can choose a probability of error in rejecting the decision of the1NN classifier of 10, 5 and 1%. If each of the features used by the 1NNclassifier passes the Grubbs' test some embodiments do an additionalmultivariate outlier test such as the Mahlanobis d-squared test. Thesestatistical outlier tests are described by Barnett and Lewis (1994). Themean and standard deviation of each feature for each class will becalculated at the time of training and will be added to the featurefile.

In some embodiments, the user chooses whether to use the extendedsilhouette-matching routine when the 1NN classifier finds ambiguity(poor statistical match) by including the optional argument “-silh”followed by the name of the prototype silhouette file on the commandline. The extended silhouette-matching classifier will increase theaccuracy of classification by either confirming that the 1NN classifierchose the correct class or it may find: that the correct class is adifferent species; the detection is clutter (no portion of the detectedarea matches one of the prototype silhouettes adequately) and report itas the class, OTHER; that the detection is a case of overlappingspecimens and it will classify each of them; or some combination of thethree previous decisions. Thus, the silhouette/color matching method isuseful for classifying detections when the 1NN classifier's resultssuggest there is some uncertainty, perhaps due to occlusion (bodies ofarthropods partially covering each other), or where parts of arthropodsare missing due to damage. Experiment 2B gives examples of how thisprocess works (FIGS. 36-37).

To keep the number of reference specimens in the data base of prototypefeature files to a manageable number while still retaining most of theinformation about the distribution of features for each class, Hart'scondensed nearest-neighbor algorithm is used (Hart, P. E. 1968. “Thecondensed nearest neighbor rule.” IEEE Trans. Inform. Theory. IT-14, pp.515-516). Hart's algorithm can reduce the number of references withoutgreatly decreasing the accuracy of the classifier.

The detection/classification results can be sent in text form either tothe user's screen or to a text file. In addition, the results aregraphically displayed using color, for rapid recognition by the user(see FIGS. 28 and 29). This results image is saved as a file called,ClassifyImg.bmp. Each detected region is labeled with the color that isassociated with the class that has been assigned to the detection. Thecolors are chosen in advance and set inside the program,BugClassify.exe. Species 1 is assigned the color green, species 2 blue,species 3 yellow, etc. The species of arthropods are assigned to thesecolor indices when the user edits the feature file and gives eachreference a species classification number. It is this number that isused to assign the color code. The species number, 0, is reserved forthe class OTHER and OTHER is assigned the color red.

An Experimental Demonstration of the Concepts

This section demonstrates practical applications of the inventions.Experiments 1 and 2 show that the technology can be configured in aversion that uses a color scanner, connected to a host computer, toacquire arthropod images. This configuration would commonly be usedindoors in laboratory and office settings to count insects and/orclassify them. For indoor use a scanner may be preferable to a digitalcamera for acquiring arthropod images, since scanners are generally lessexpensive than a camera of comparable color quality and resolution. Inaddition, a scanner is able to image a larger area than a camera, whichis beneficial for processing samples containing many arthropods.Furthermore, a scanner, unlike a digital camera system, does not need asupplemental light source to insure uniform lighting. A light source isalready incorporated in the scanner, making system integration muchsimpler.

These first two experiments were conducted to illustrate the usefulnessof the technology to a wide variety of users such as environmentalscience and biology teachers, ecologists, entomologists, pest managementspecialists and custom inspectors. These professionals can use thetechnology for the following: a) students in an ecology class collectinsects and want to rapidly classify them. The insects are collected,killed and placed on a scanner in one of the embodiments configured forthis application; b) an insect-pest specialist takes samples of insectsin their habitat using sampling devices such as sweep nets, aspiratorsor D-Vacs (a vacuuming device), optionally kills, immobilizes or knocksthem out by chemical means, and then deposits the sample on the surfaceof the scanner to have them automatically classified and counted; and c)a county agent who classifies arthropods as a community service or acustom's agent in charge of classifying insects in luggage rapidly killsthe insect with a kit provided with the system, and places the unknownspecimen on the scanner surface for classification. The system wouldcompare the unknown insect against one of the databases of knownprototype insects.

Some embodiments are designed to be highly customized for specificapplications. For example, in the case that an embodiment is for acustoms facility, the system would be configured in such a way that theuser has in the system's database of prototype features and silhouettes,insects of relevant importance to the concerns of that particularcustoms office.

In another embodiment, when logged on to a host personal computer (PC),the user places insects or other arthropods on the scanning surface,acquires images of those specimens and stores them on the host computer.The user employs an embodiment of the invention's software on the hostPC to detect, classify and count the arthropods that were placed on thescanner.

Experiment 3 demonstrated an alternative approach to collecting andprocessing images of insects and arthropods. Rather than obtaininginsect images by placing them on the surface of a scanner, a digitalcolor camera is placed near and with a view of the arthropod-collectingor -detection surface. The portability and small size of a camera asopposed to a computer scanner is appropriate for field conditions,especially as part of automatic sampling devices. A digital camera isalso preferred for the hardware portion of the system when magnificationvia a lens is needed. Experiment 3 shows that a camera-based system canautomatically detect, classify and count insects that have been caughton or in traps in the field, or after being collected have been placedon another type of detection surface.

Experiment 1: Equipment Setup. See FIG. 6.

1.—An Epson Perfection 1200U scanner communicated with a Macintosh PowerMac G4 (Mac OS X Version 1.5 operating system) via a Universal SerialBus (USB) connection. The scanner used the TWAIN 5 software. Thissoftware allows the user to collect images and adjust image quality. TheTWAIN software initially shows a preview image of the entire scanningsurface. A portion or subwindow was then selected that included all ofthe insects/arthropods, before the final full-resolution image wasrequested. The resulting images were saved to a bitmap format file forfurther processing. The scanner has an imaging surface that is 21.6 by27.9 centimeters in area (8½ by 11 inches) and can collect images ofresolutions ranging from 50 to 9,600 dpi. The scanner can save the coloras 24 or 48 bit information. A spatial resolution of 96 dpi and 24-bitcolor were used. To avoid crushing the insects on the scanner's glasssurface with the scanner's cover or lid, a white cardboard box was usedas a cover and background. The box was 19 cm wide, 28 cm long, and 5.1cm high (7½×11×2 inches).

2.—Adobe Photoshop 7.0 software on the Macintosh was used to collect theimages of insects placed on the scanner's surface. Adobe Photoshophanded off control for an image collection request to Epson's TWAIN 5software.

3.—The image files were transferred to a Dell Dimension XPS T550 PC(Intel Pentium III processor) over the internet by attaching each imagefile to an email message. The image-processing software was executed onthis Dell PC. The Dell utilized the second edition of the MicrosoftWindows 98 operating system. Two different computers were used becausein the particular embodiment employed in Experiment 1 theimage-processing software only ran on a PC using a DOS shell of any ofthe Microsoft Windows operating systems, but the scanner was connectedto and set up for a Macintosh computer. In many embodiments, the systemwould be configured in a manner that the scanner and processing softwarewould all be hosted by just one computer.

Description of the Experiment and its Results

For clarity, the same sequence of functions described above in thesection, GENERAL DESCRIPTION OF THE OPERATION OF THE SYSTEM, is usedhere to describe how this configuration of the technology worked forthis experiment. The system was configured to simulate a situation whereit is used to classify insects that are collected in a habitat ofparticular interest. For example, an ecology instructor wants to use thesystem to assess the abundance of certain insects in a horticulturalgarden at various intervals. Thus, insects from an urban Minneapolis,Minn. garden were used to simulate this example.

A. Generation of a Background Image

Normally when the system is used, the first function is to collect abackground image, that is, an image of the detection surface 624 priorto placing insects on the surface. However, for applications like this,some embodiments do not need to generate a background image. Thisexperiment demonstrated that the software does not require a backgroundimage as input. In some embodiments, the system can estimate theappearance of the detection surface without insects from the backgroundof the image with insects. Each pixel of the estimated background usedthe median pixel values for the color components R, G, and B, from thetest image. See explanation in “General Description of Operation of theSystem”, function A.

B. Generation of Identifying Reference Features from Known Arthropods

In this function features were extracted from a set of images containingidentified and representative insects collected in the garden. Eleventraining or reference insects were used, which included 6 species:

-   -   1) two individuals of a species of syrphid fly that has yellow        stripes on its thorax (Diptera: Syrphidae);    -   2) two asparagus beetles, Crioceris asparagi (Linne)        (Coleoptera: Chrysomelidae);    -   3) one individual of a second syrphid fly species with no        stripes on its thorax. It appears to mimic a honey bee (Diptera:        Syrphidae);    -   4) three halictid bees (Hymenoptera: Halictidae);    -   5) one blow fly (Diptera: Calliphoridae);    -   6) two multicolored Asiatic ladybird beetles, Harmonia axyridis        Pallas.

The features were generated by executing the detection andclassification software, called BugClassify.exe, in what is referred toas the “training mode,” with the image of the known prototypes orreference insects as input. In the “training mode” the software executesin exactly the same manner as in the “detection/classification mode”until the last function, classification. Instead of trying to classifythe insects, in the “training mode” the software saves the feature setassociated with each known insect to a file called the feature file(Identifying Reference Feature file). Once this file is generated itmust be edited by the user by adding the species name and species codenumber to each reference insect's set of features. A code number of theprototype's aspect or orientation is also added. This allows theclassifier to assign the species name or identity associated with thefeature set that best matches the features of the unknown.

In some embodiments, the present invention is successful even atdistinguishing different color forms of a single species of ladybirdbeetle, the multicolored Asiatic ladybird beetle.

The scanning system was used to acquire two images containing referencespecimens of garden insects. These two pictures contained the same 11individuals. In one image the insects were placed with a view of theirdorsal surface while in the other they were oriented with a view oftheir ventral surface. In a few cases the insect's legs or wingsinterfered with getting a true dorsal or ventral view. In such cases,these insects had a portion of their lateral side also in view.

The two training images were collected on the Macintosh via AdobePhotoshop. In Photoshop's main window the File menu was clicked with themouse and Import Epson Scanner Enable was selected. This brought upEpson's TWAIN 5 software which does an initial pre-scan. A subwindow wasselected for the final image, color photograph was selected for theimage type, 96 dpi was selected and then Scan was clicked. After eachimage was captured and displayed, Save As was clicked, the name of thefile was entered and then the Save button was hit.

The two images were saved as two files, ScanDorsalTrain.bmp (FIG. 25)and ScanVentralTrain.bmp (FIG. 26). BugClassify.exe was executed witheach of these images as input to generate a reference feature file. Thefunctions were as follows:BugClassify ScanDorsalTrain estimatebackground ScanDTrain-trainandBugClassify ScanVentralTrain estimatebackground ScanVTrain-train

The two resulting feature files, ScanDTrain.txt and ScanVTrain.txt, weremerged into one file, ScanTrain.txt, in the text editor, CodeWright, andthe species identification for each feature set was also added inCodeWright.

Although the software BugClassify.exe calculated all the statisticalfeatures mentioned in Section E of the previous section, “GeneralDescription of the Operation of the System,” only seven of the featuresare chosen, in this embodiment, to be saved for identification to thefile, ScanTrain.txt. The seven features were:

-   -   Size-related features:        -   1) total area;        -   2) perimeter;    -   Shape-related features:        -   3) Circular fit or compactness feature—sometimes referred to            as the isoperimetric quotient, defined as 4(pi) times the            total area divided by the square of the perimeter (Russ            1995). This feature is used to measure how close to a circle            and how compact an arthropod's shape is.        -   4). Rectangular fit feature—this feature calculates how            close an insect is to a rectangle in shape.    -   Luminance features:        -   5) Average Intensity Difference—the average of the            difference in intensity between the object and its            background. As long as the lighting is controlled, keeping            it nearly constant, this feature provides information about            the relative amount of light that the object reflects.        -   6) Coefficient of Variability in Intensity Difference—the            relative amount that the intensity difference varies over            the object. This feature is calculated by dividing the            standard deviation in the intensity difference (difference            between object and background) by the mean intensity            difference.    -   Color feature:        -   7) Color feature matrix—the 2D hue/saturation color            histogram that was developed and which provides a simple and            practical way to summarize the color of an arthropod or            insect that is independent of scale and rotation in the            image.

Prototype silhouettes were not extracted for this experiment, to showthat the nearest-neighbor classifier works well on its own without theextended silhouette-matching method (See Section B in “GeneralDescription of the Operation of the System” for a complete listing ofthis embodiment's capabilities).

With the feature file, ScanTrain.txt, the system was then configured toidentify unknown insects. The feature file contained the feature setfrom the 22 insect images shown in FIG. 25 and FIG. 26. The 22 insectimages were actually a dorsal and ventral view of 11 individual insectsrepresenting 6 species.

C. Acquisition of Images of the Unknown Arthropods to be Detected

Two pictures were collected to test the ability of the equipment,process and software to detect and recognize various insects. Theseimages simulate the actual use of the scanner-based system fordetecting, identifying and counting insects. Each image used 10 insectsthat had not been used to train the system. The 10 insects included: 1)two syrphid flies of a species with a striped thorax; 2) one syrphid flyof the species without a striped thorax; 3) two halictid bees; 4) oneblow fly; 5) two Asiatic ladybird beetles; and 6) two asparagus beetles.

The two pictures (FIG. 27A and FIG. 27B) were taken of the same 10individual insects. The insects were first placed with their dorsal sidedown on the surface of the scanner. An image was captured and saved as acomputer file, ScanDorsalTest.bmp (FIG. 27A). The next function was toplace the same insects with their ventral side on the scanner's surface.They were scanned and this second image was saved as a computer file,ScanVentralTest.bmp (FIG. 27B). The insects in each image were placed atvarious angles of rotation in the 2D image space to show that the systemis insensitive to rotation.

D.-F. Arthropod Detection, Feature Extraction, Classification

In this function the system labeled each pixel from the test images ofthe insects to be identified (FIG. 27A and FIG. 27B) that appeareddifferent from their corresponding background images, and thus werelikely to belong to an insect or clutter. The labeled pixels were thenconnected into continuous regions or blobs by connected-componentsanalysis. Regions that were too small in area were discarded. Featureswere extracted from each detection (i.e., each detected object) andthese features were then compared with the feature set of each known orreference insect via the single-nearest-neighbor classifier. Althoughthe feature file contained the values for the seven previously describedfeatures, the classifier was instructed to use just four of thefeatures: area, circular fit, average difference in gray level orluminance and the invention's color feature.

The first test image, ScanDorsalTest.bmp (FIG. 27A and FIG. 28A) wasanalyzed by running the executable software, BugClassify.exe, with thisimage and the feature file, ScanTrain.txt (generated in function B), asinput. FIG. 28B has the output image from that process. All 10 insectswere detected with no false detects. Each of the test insects matchedwell with a reference of the correct species so it was not necessary toassign any of them to the unknown class, OTHER. BugClassify.exe outputto the computer screen a summary of the numbers counted for eachspecies, but that is not shown here. The detected or labeled pixelsassociated with each of the detected insects were replaced in the outputimage with the color code for the species class that was assigned by theclassifier. Non-detected pixels in this output image were assigned thesame values as they had in the input image. The color code is asfollows:

-   -   GREEN=yellow striped thorax syrphid fly;    -   BLUE=orange non-striped thorax syrphid fly;    -   YELLOW=asparagus beetle;    -   ORANGE OR BROWNISH RED=halictid bees;    -   LIGHT BLUE GREEN=blowfly;    -   PURPLE=Multicolored Asiatic Ladybird beetle.    -   RED=OTHER

Here is a brief explanation of how the system identified the unknowninsects. The classifier in the software calculated the percentagedifference in each feature with respect to the prototype's value. Thus,the percentage difference was calculated between the unknown and knownfor area, shape, and luminosity, as well as the percentage difference inoverlap of the two color matrices. The classifier then used each ofthese “normalized” features (feature is scaled by its expected value,which is the reference specimen's value) to generate an overallgoodness-of-fit measure. This goodness-of-fit measure is a Euclideandistance metric, the square root of the sum of the squares of thepercentage difference in each feature. The unknown was assigned to theclass of the prototype with the smallest or closest value for thisEuclidean distance. If, however, the best match differed by more than40% in area, or if the contents of the two 2D hue/saturation histogramsoverlapped by less than 68%, or the average gray-level difference wasoff by more than 12%, or the overall Euclidean metric differed by morethan 1.0 (fraction rather than a percentage), the conclusion was thatthe match was not good and that the object must be something that hadnot been presented to the classifier during training (a species orobject not represented among the prototypes of the feature file). Inthis case, the unknown was assigned to a class called OTHER. Thesethresholds were empirically arrived at by prior testing with severalsets of other types of insects and additional images of the same typesof insects used in this experiment. Note that a poor match can alsoindicate that there is more than one arthropod and that one is occludingthe other or that the individual arthropod may be damaged or unusual insome other way. These possibilities were addressed in Experiment 2B.

The processing of the second test image (FIG. 27B and FIG. 29A) alsoproduced correct results (FIG. 29B). All the insects were detected andcorrectly identified without any false alarms. The inputs to this testwere the input file, ScanVentralTest.bmp (FIG. 27B and FIG. 29A) and thereference file ScanTrain.txt. Note that for both test images shadows,particularly those associated with the larger flies, did not cause anyproblems. They were not detected or segmented along with the insect. Theimage-processing algorithm is able to recognize shadows and thus avoidslabeling shadow pixels as being significant from the background.

To summarize Experiment 1, the validity and practicality of theinvention's concepts were demonstrated. It was shown that the inventionis able to detect insects within an image and avoid detecting shadows orincluding them with the labeled area of the insect. It was shown that itis possible to generate distinguishing features to recognize insects. Itwas also demonstrated, that the orientation of the insect or arthropodis not critical to its identification, provided that there is a distinctset of features associated with each position and that the insect andits position are represented among the prototypes of the feature file.Finally, it was also shown that by using the invention'simage-processing algorithms, a color computer scanner and a computersystem, it is possible to automate the detection and classification ofinsects.

Experiment 2A.

This experiment builds upon what was done in Experiment 1 and wasperformed to prove that the nearest-neighbor classifier is able todistinguish between the insects it has been trained to recognize andvarious forms of clutter that could be present in some applications. Forexample, if the user placed insects along with plant parts on thescanning surface. This could happen if an embodiment of the system isused by a person who is sampling insects on vegetation with a sweep net.The sampler sweeps the net over vegetation that may harbor insects,transfers the collected material that includes insects and plant partsto a device to kill the insects and then places the collected materialon the scanning surface of the system. In addition, it was shown thatthe feature set and 1NN classifier are robust, since they can oftenidentify incomplete arthropods, i.e., insects or arthropods with partsof their body missing from the damage caused by handling them after someof them had become dry and brittle.

-   -   Equipment setup. FIG. 6. Same setup as in Experiment 1.        -   1.—An Epson Perfection 1200U scanner connected to a            Macintosh Power Mac G4 via a USB cable, to collect the test            image.        -   2.—Epson's TWAIN 5 software via Adobe's Photoshop 7.0 was            used to set the scanner's resolution to 96 dpi with 24-bit            color resolution and to request an image.        -   3.—The collected images were processed with the            image-processing software on a Dell personal computer with            an Intel Pentium III processor running the Microsoft Windows            98 operating system.

Description of the Experiment and its Results

The same sequence of functions described above in the section, GENERALDESCRIPTION OF THE OPERATION OF THE SYSTEM, is used here to describe howthe system identified and counted several insects that were mixed withplant material (clutter). This was done to show that the system canreject objects that are not the arthropods that the classifier has beentrained to recognize. The contents of a sampling tool, such as an insectnet, may deposit vegetation and other debris on the detection surfacebesides arthropods. The insects were collected from vegetation in aMinneapolis garden.

A. Generation of a Background Image

As in Experiment 1, a background image of the scanner's surface was notcollected before the insects were placed on it. The system estimated abackground (see explanation in Section A of Experiment 1) from the testimage.

B. Generation of Identifying Reference Features from Known Arthropods

Since this experiment worked with the same insect species imaged underthe same scanner conditions as in Experiment 1, the system was alreadyconfigured for this situation. The computer contained the feature filethat was generated for Experiment 1. This file contained the followingseven features for each reference specimen: 1) area; 2) perimeter; 3)circular or compactness shape feature; 4) rectangular shape feature; 5)average difference in luminosity between the insect and the background;6) the relative variance in the average intensity difference; and 7) thecolor feature or 2D hue/saturation color histogram. This feature filecontained the feature sets of the 22 insect images shown in FIGS. 25 and26. The images were dorsal and ventral views of 11 individual insectsrepresenting 6 species. Prototype silhouettes were not generated forthis experiment, to demonstrate that the 1NN classifier can recognizeand reject clutter.

C. Acquisition of Images of the Unknown Arthropods to be Detected

The next function in this demonstration was the simulation of placing amixture of insects and plant parts on the scanning surface. Seveninsects mixed with plant material were dropped on the scanner's surfaceso they would appear in various natural and “random” orientations whichmight be typical of emptying insects from a sampling device. The seveninsects included: one striped-thorax syrphid fly, one blow fly, twoAsiatic ladybird beetles, and two asparagus beetles. The plant materialor clutter that was placed on the scanner surface included: 1) one sugarmaple seed (Acer saccharum Marsh); 2) one Amur maple seed (Acer ginnalaMaxim); 3) one green ash seed (Fraxinus pennsylvanica Marsh); 4) a shootof Korean boxwood (Buxus harlandii Hance); and 5) two fragments ofbluegrass (Poa pratensis L.). As an additional challenge for thesystem's ability to identify arthropods, two of the insects in this testcase were significantly damaged. The syrphid fly (top of FIG. 30A) wasmissing its abdomen and the asparagus beetle (bottom of FIG. 30A) had nohead and thorax. In a real world application, damaged specimens might beexpected even though precautions should be taken in the handling of thearthropods to increase the accuracy of the system.

An image was acquired (FIG. 30A) and saved as a file, as was describedin Experiment 1, Section B. This image was saved as a file calledScanClutter5.bmp.

D.-F. Arthropod Detection, Feature Extraction, Classification

As indicated for Experiment 1, this function involves: 1) labeling eachpixel from the test image that appeared different from the correspondingbackground image and thus is likely to belong to an insect or clutter;2) connecting the labeled pixels into continuous regions or detections;3) extracting features from the detections; and 4) classifying thedetections by comparing their features with those of the known insectsin the input feature file. As in Experiment 1, the classifier was set touse only four of the seven features in the feature file: area, circularfit, average difference in gray level or luminosity and the 2Dhue/saturation color histogram. Each unknown or detection was assignedto the class of the prototype with the shortest Euclidean distance inthe four dimensional (four features) feature space. However, if thisdistance was greater than 1 (fraction, same as 100% difference), or ifthe difference in area between the unknown and best match was greaterthan 40%, or if the average gray-level difference between the two wasmore than 12%, or if the two 2D color histograms overlapped by less than68%, it was concluded that a good match was not present. Thus for a poormatch, the object was assumed to be something that had not beenpresented to the classifier during training. In this case the unknownwas placed in the undetermined class, OTHER.

The software program, BugClassify.exe, was executed withScanClutter5.bmp (image of FIG. 30A) and the feature file, ScanTrain.txtas input. The following command was used:BugClassify ScanClutter5 estimatebackground ScanTrain.

The output result image that was obtained appears on FIG. 30B.BugClassify.exe also sent a listing to the computer screen of theclassification results for each object that was detected and listed asummary of the numbers detected for each class. The class assigned toeach detection was colored coded in the output image as in Experiment 1:

-   -   GREEN=yellow striped thorax syrphid fly;    -   BLUE=orange non-striped thorax syrphid fly;    -   YELLOW=asparagus beetle;    -   ORANGE OR BROWNISH RED=halictid bees;    -   LIGHT BLUE GREEN=blowfly;    -   PURPLE=Multicolored Asiatic Ladybird beetle.    -   RED=OTHER

The seven insects detected and correctly identified included the syrphidfly and asparagus beetle that were missing a significant portion oftheir bodies (FIG. 30B). This illustrates how robust thenearest-neighbor classifier is because it uses a set of complementaryfeatures. Missing an abdomen or head may produce a misleading size orshape feature, but the color and luminance features may still beadequate for good classification. The six pieces of plant parts weredetected but rejected as not being relevant to the sampling goals. Theywere labeled as red in the output image and as OTHER in the textoutput's summary. While the grass and ash seed were each detected as oneuniform region, the boxwood foliage and maple seeds were each detectedas separate multiple regions, but all of these regions were rejected asclutter. Note that in this test case (FIG. 30A and FIG. 30B), as in theprevious experiment, the shadows in the images did not cause anyproblems. As may have been noticed based on the name of this test image,ScanClutter5.bmp, there were four other similar images, each withdifferent insects and arrangements of plant parts. The four other testcases were completely successful at detecting and identify insects aswell as rejecting plant material. The case for ScanClutter5.bmp wasincluded here as it was the most complicated of this test series.

Experiment 2A provides another demonstration of the validity andpracticality of the concepts of some embodiments. Some embodiments areable to detect insects within an image. With the statistical featuresthat the software extracts and the nearest-neighbor classifier that usesthese features, insects are recognized that are included in thetraining/feature file. Objects that were not intended to be detected andcounted—clutter—were appropriately assigned to a class called OTHER orunknown. It was again demonstrated that both the detection andclassification of arthropods can be automated.

Experiment 2B.

This experiment demonstrated the versatility and strength of the systemsto identify insects even when they overlap (occlusion). Dealing withocclusion can be important. While a user who places his arthropods on ascanner for counting and identification always has the option of makingsure the insects don't overlap or touch one another in order to insuregreater accuracy (as in Experiment 1 here), this will not always bepossible. It will not be possible to prevent occlusions when embodimentsof the systems are configured to include unattended insect-monitoringdevices in the field, such as sticky traps. In this situation, a stickysurface, where insects are trapped, will be scanned by an imaging deviceand the resulting images analyzed. As insects accumulate over time theywill overlap (occlusion). The demonstration here illustrates that thesoftware has two ways to deal with overlap in arthropod specimens: 1)subtracting from the occlusion arthropods that were previously detectedin earlier image collections. This approach assumes that the system isconfigured to trap and monitor insects periodically over time, so thatthe earlier of the overlapped insects are known and can be subtractedout along with the background; or 2) using the higher-level extendedsilhouette-matching classifier in conjunction with the lower-levelnearest-neighbor classifier to solve the ambiguity. For more informationon silhouettes see Section B. This experiment demonstrated that thenearest-neighbor classifier that is utilized is robust and can recognizecomplex situations like occlusion or difficult clutter, and request thatthe extended silhouette-matching method confirm its identifications orhave the, silhouette matching do further analysis on difficult cases.

The same sequence of functions described above in the section, GENERALDESCRIPTION OF THE OPERATION OF THE SYSTEM, which was previously used indescribing Experiments 1 and 2A, is also used here to describe how thesystem detected and identified occluded insects:

A. Generation of a Background Image

A background image of the scanner's surface was collected with insectsbut before additional insects were placed on it. This was done to showthe advantage of using a previous image as a background image in thecase of occlusions rather than estimating the background as was done inExperiments 1 and 2A. The events were simulated that would occur if anembodiment of the system was configured as a monitoring device thatcollected images at periodic intervals. First an image that wascollected at an initial period was simulated. For this a backgroundimage was collected of the scanner's surface with insects and clutter(plant material), by placing two insects and a plant seed on thescanning surface. This image was saved as the file, Occ1A.bmp (FIG. 31).

B. Generation of Identifying Reference Features from Known Arthropods

For this experiment it was not necessary to generate a reference featurefile because this was done in Experiment 1. Thus, the feature file wasused, ScanTrain.txt, generated by that experiment as input for thistest. For this application, the system's silhouette-matchingcapabilities also were used. Therefore, prototype silhouettes had to begenerated. The prototype silhouettes were extracted from among thereference insects in the training image, ScanDorsalTrain.bmp (FIG. 26A),of Experiment 1. The silhouette from one individual of each of the sixinsect species was used. The silhouettes were taken from the followinginsects of ScanDorsalTrain.bmp:

-   -   1) the syrphid fly with the striped thorax was represented by        the silhouette of the top left-most insect;    -   2) the asparagus beetle's silhouette was from the right-most        asparagus beetle of the second row;    -   3) the syrphid fly without a stripe was the right-most insect of        the second row; 4) the right-most halictid bee in the third row        was used for a silhouette;    -   5) the blow fly silhouette was extracted from the blow fly of        the fourth row; and    -   6) the ladybug silhouette was taken from the right-most ladybird        beetle.

The following process was used to extract the silhouettes. Since thesilhouette-generating and -matching routines were not completelyintegrated into this embodiment's overall program, BugClassify.exe, atthe time of this test, the silhouettes were generated by a series ofcommands. First, the program BugClassify was executed using the fileScanDorsalTrain.bmp as input along with the arguments to tell thecommand to operate in the training mode. In a DOS shell the followingcommand was typed:BugClassify ScanDorsalTrain estimatebackground trainingmode

One of the intermediate outputs from this command is the labeled image,LabelImg.bmp, which contained the labeled detections. Another command,MakeSilh.exe, was executed with LabelImg.bmp as input and SilhImg.bmpwas the output. The command line in DOS for this function looked likethis:MakeSilh LabelImg SilhImg

The latter image contained the silhouette images of the referenceinsects. Finally, SilhImg.bmp was used as input for GetSilhCode.exewhich used the silhouette images to generate the prototype silhouettesin a chain-code form which was saved to a file calledScanSilhouette.sil. The command was as follows:GetSilhCode SilhImg ScanSilhouette

This file was hand edited in the text editor, CodeWright, to append thecolor reference points to each of the six desired silhouettes. The extrasilhouettes were deleted. The silhouettes are illustrated in FIGS.32A-32F.

C. Acquisition of Images of the Unknown Arthropods to be Detected

The next function was the simulation of insects that are occluded. Foroverlapping pairs of insects, the following were placed on thescanner: 1) a pair of asparagus beetles abutting one another with littleor no overlap; 2) a pair of multicolored Asiatic ladybird beetles wereplaced side by side with little overlap; and 3) an ash seed waspositioned so that it obscured at least half the view of a halictid bee(FIG. 33).

An image of these occluded insects was acquired in the same manner asdescribed in Experiment 1, Section B. The image was saved as a filecalled Occ2A.bmp. This file was created to show that in the case ofocclusion the nearest-neighbor classifier can correctly identify membersof an occlusion if the system has information about one of the membersof the occlusion from a previously processed image. Otherwise it was tobe demonstrated that the occlusion problem can still be solved by thenearest-neighbor classifier's calling upon the higher-level classifier,the extended silhouette-matching method.

D.-F. Arthropod Detection, Feature Extraction, Classification

The program BugClassify was run with Occ2A.bmp as the current imageinput file, Occ1A.bmp as the previous image input file, andScanTrain.txt (from Experiment 1) as the feature file. The command lineappeared in DOS as follows:BugClassify Occ2A Occ1A ScanTrain

All three insects added since the collection of the image Occ1A weredetected and correctly identified (FIG. 34). By taking the difference inthe luminance and color between the two input images, the algorithmdetected only the objects that were new to Occ2A.bmp and had not been inOcc1A.bmp. This left unambiguous detections for the asparagus andladybird beetles. The nearest-neighbor classifier found good matches forboth these detections since the complete insects were detected. Thenearest-neighbor classifier even found the halictid bee was the bestmatch for the occluded bee in spite of the fact that half of it wasmissing from view. Even though a halictid bee was the best match for theoccluded bee, its matching score was poor enough to make thisidentification uncertain. The nearest-neighbor classifier was able toselect the halictid bee because prior to running this test the set offeatures used by the classifier in Experiments 1 and 2A was changed.Three of the four previously used features were used, area, averagerelative luminance and the color feature, but the roundness feature wasreplaced with the insect's width. It was known in advance that the shapeof the bee would be compromised by the occlusion, but not the bee'swidth. Although the best match for the occluded bee was a halictid bee,it was a poor match with respect to the area, color and gray-levelfeatures. By the criteria or threshold set in Experiment 2A forrejecting something as clutter, the occluded bee was considered aspossible clutter and was left for the silhouette-matching method toclarify. The matching scores of the beetles, on the other hand, weregood enough to accept without further analysis.

Although integration of the silhouette routines into BugClassify had notbeen finished at the time of this experiment, manual simulationsdemonstrated how the nearest-neighbor classifier will interact with theinvention's color extended silhouette-matching classifier, in someembodiments. Since the nearest-neighbor-classifier matching scores forthe asparagus beetle and ladybird beetle were very good, there was noneed to invoke the silhouette classifier to confirm theiridentification. Where the matching metrics of the nearest-neighborclassifier indicated there was a good match, silhouette matching isoptionally omitted in this embodiment, since it is currently acomputationally-intensive and time-consuming method. The followingfunctions were taken to simulate how the software will process the caseof the occluded bee. As described in Section B of this experimentMakeSilh.exe and GetSilhCode.exe were used to generate a silhouettechain code file of the halictid bee from the intermediate label imageproduced by BugClassify. TransSilh.exe then read in the silhouette codeof the occluded bee and the prototype silhouettes in ScanSilhouette.sil.The command in the DOS window appeared as follows:TransSilh HBeeSilh ScanSilhouette Occ2A

TransSilh placed each prototype silhouette with its center overlappingthe center of the occluded bee's silhouette. It then rotated eachprototype 360 degrees at one degree increments. At each increment ofrotation it tried shifting the prototype silhouette by as much as 30pixels in both directions of x and y. The best matches were recorded foreach of the prototypes. TransSilh then assigned the occluded bee to theclass of the prototype that had the best match, provided that enoughpixels of the silhouettes overlapped and the pixels for color samplingagreed with those of the unknown. The halictid bee's prototypesilhouette matched the occluded halictid bee best (FIGS. 35A-35C). Thiswas considered an acceptable match, as nearly half the occludedsilhouette's pixels overlapped those of the prototype and its remainingpixels were close to those of the prototype and three of the six colorsample pixels matched those of the prototype. If this best match hadaccounted for only a portion of the occluded bee's area, the methodologywould have continued considering the other good matches for theremaining portions of the detection, as there could have been anotherinsect that was part of the detection. In the case of the occluded bee,the bee prototype accounted for the entire area of the unknown bee. Thisportion of the experiment demonstrates that the extendedsilhouette-matching routine can be useful for correcting or confirmingidentifications by the nearest-neighbor classifier. It was alsoillustrated that having information about previously trapped insects canaid in solving occlusion problems simply by subtracting the previousimage from the current one.

One more test was conducted as part of this experiment to demonstratethat the nearest-neighbor classifier can detect a matching problem foreach of the three occlusions and request that the extendedsilhouette-matching routine do further analysis. For this, a backgroundimage that contained insects was not used. Thus, in this test the systemhad no prior knowledge about one of the members of each occlusion. Theprogram BugClassify.exe was executed with the image, Occ2A.bmp (FIG. 33)and the feature file, ScanTrain.txt as input.

The command appeared in the DOS shell as follows:BugClassify Occ2A estimatebackground ScanTrain

BugClassify.exe estimated a background from the test image containinginsects (FIG. 33). BugClassify detected all three sets of occludedinsects without any false alarms (FIG. 36). To generate FIG. 36, aversion of BugClassify was executed that outputs a decision image thatcolor codes each detection with the best match of the nearest-neighborclassifier, regardless of whether the classifier would eventually rejectit as possible clutter. If the normal version of BugClassify had beenused, it would have reported all three detections as OTHER and coloredthem red. This version of BugClassify was used to simulate theinvention's approach of having the nearest-neighbor classifier withholdfinal judgment and pass the final decision to the extendedsilhouette-matching classifier. FIG. 36 shows that the nearest-neighborclassifier found the best match in statistical-feature space for thepair of asparagus beetles was a blow fly. This figure also displays thatthe pair of ladybird beetles and the ash seed with halictid bee bestmatched a syrphid fly. The feature-matching scores for the best matchfor each of these three detections were sufficiently poor to suggestthat they could represent either clutter or occlusions. Thenearest-neighbor classifier rejected the best matches as acceptablebecause the clutter-rejection criterion that was mentioned in Experiment2A was exceeded in each case. The color feature and gray-level featurewere too dissimilar to have confidence in the best match. In addition,the area of the ash seed with the bee was far too large to actually bethe best match, a syrphid fly.

The action of the extended silhouette-matching routine was simulated byexecuting the following sequence of functions:

As was mentioned in the previous paragraph, the command BugClassify wasexecuted with the option to estimate the background. BugClassify inaddition to producing the output image of FIG. 36 also produces anintermediate results image called LabelImg.bmp. This is a labeled imageof the detected areas after the connected-components software hasgrouped the pixels that appear different from the background, intocontiguous regions. LabelImg.bmp was used as input to the command,MakeSilh.exe. MakeSilh.exe produced an image with silhouettes of theunknowns called OccSilh.bmp. The command line in DOS appeared asfollows:MakeSilh LabelImg OccSilh

The command, GetSilhCode.exe, was then used with OccSilh.bmp as input togenerate a chain-code silhouette file called TestOccSilh.sil as follows:GetSilhCode OccSilh TestOccSilh

In the text editor, CodeWright, each of the chain codes for the threeoccluded detections was copied to their own silhouette-chain-code filescalled: TestABSilh.sil, TestLBSilh.sil and TestHBSilh.sil. These threechain-code files contained the silhouette chain code for the asparagusbeetles, ladybird beetles and ash seed/bee, respectively. The prototypesilhouette file, ScanSilhouette.sil, and the command, TransSilh.exe,were used with each of the occluded silhouette files to find the bestmatches for each occlusion and to simulate the higher-levelclassification logic. To do this the following three commands in the DOSshell were run:TransSilh ScanSilhouette TestABSilh Occ2ATransSilh ScanSilhouette TestLBSilh Occ2ATransSilh ScanSilhouette TestHBSilh Occ2A

The extended-silhouette-matching method correctly detected andidentified each of the beetles. For the detection that included the pairof asparagus beetles, the best match was for the asparagus beetle on theleft (FIG. 37A). This best match was the prototype of the asparagusbeetle. Thus the beetle on the left was accepted as an asparagus beetlebecause: 1) more than half the pixels of the prototype's silhouetteoverlapped the silhouette of the unknown; 2) the remaining pixels of theprototype's silhouette were a short distance to the unknown'ssilhouette; and 3) more than half the sample pixels for color matchedthe unknown's corresponding pixels in color. If more than 40 to 50% ofthe prototype's silhouette overlap the unknown's silhouette (or viceversa) and half or more of the color sample pixels agree with theunknown in color, then it is considered that the match can be acceptedas correct and the class of the prototype can be assigned to theunknown. The best match in a remaining portion of the asparagus beetleocclusion was also the prototype silhouette of an asparagus beetle (FIG.37B). This match was also accepted as a correct identification becausenearly half the pixels of the silhouette prototype overlapped theoccluded area's silhouette, and most of the color sample pixels agreedin color with those of the unknown.

The identification process for the ladybird beetles was similar to thatof the asparagus beetles. The best match and second-best match for theladybird beetle detection was the prototype silhouette of the ladybirdbeetle. The prototype ladybird silhouette and the ladybird in the lowerright produced the best match (FIG. 37C), while the second-best matchwas between the prototype silhouette of the ladybug and the silhouetteregion corresponding to the ladybug in the upper left (FIG. 37D). Bothof these matches were considered correct identifications since half ormore of the prototype's silhouette overlapped the silhouette of theunknown and the color sample pixels agreed in color with the pixels ofthe unknown.

The invention's approach to silhouette and color-pattern matching foundthat a halictid bee was the best overall match for the area around theoccluded halictid bee (FIGS. 37E-37F) while the remaining portion of theocclusion was considered clutter. However, the matching score was nothigh enough to say with certainty that there was a halictid bee there.The match between the halictid bee prototype and the occluded bee wasthe third best in terms of percentage of silhouette pixel overlap (FIGS.38A-38C), approximately 21% of the pixels overlapped the unknown'ssilhouette, but it was the best overall match because half the colorsample pixels were correct. The remaining top silhouette matches(spurious correlations of the halictid bee and asparagus beetle with theash seed) were rejected because none of the sample pixels for colormatched the unknown area's color and the percentage of the prototype'spixels that overlapped the unknown silhouette was also low. Thus, theregion associated with the ash seed was rejected as clutter by theextended silhouette matching.

If the prototype silhouettes had been scaled (made slightly larger andsmaller) in addition to translating (shifting them in x and y, parallelto the x and y axes) and rotating them when looking for a better match,a better matching score is obtained in some embodiments, between the beeprototype and the occluded bee. This would have made the identificationof the occluded bee more certain. Although some embodiments onlytranslate and rotate the silhouettes, in other embodiments, it isadvantageous to also scale the prototype silhouettes, in order to takeinto account the natural variation in size among individuals of a givenspecies. Whether or not the classifier should count the detectedhalictid bee depends on how much uncertainty the user is willing toaccept. If the user is willing to lower the acceptance thresholds tocount this detection as a bee it is possible that the user will getadditional false detections and incorrect identifications on otheroccasions. One additional point with regard to silhouette matching basedon the occluded bee is that it may be difficult in general to recognizearthropods with much confidence when half or more of the specimen is notvisible. Clearly, if the insects are just touching or barelyoverlapping, the syntactic-silhouette-matching method can effectivelydetect and identify the members of the occlusion. This is also true ifthe older member of the occlusion can be subtracted by using a previousimage as a background input image.

Experiment 2 demonstrates that not only can some embodiments of theinvention automatically detect and identify a variety of arthropods atwidely differing orientations but they can also deal with such difficultproblems as recognizing objects that can be considered clutter, detectand count occluded arthropods and recognize arthropods with missingstructures due to damage or occlusion. It was also shown that by usingthe image-processing algorithms, a color computer scanner and a computersystem, it is possible to automate the detection and classification ofinsects for teachers, researchers, pest-management practitioners, andthe employees of various governmental regulatory agencies and publicservice departments (such as the agricultural extension service). Thisautomated technology reduces the time and cost of sampling, which willallow research and pest-management personnel to improve their monitoringof arthropod populations. With more time and lower costs, they will beable to sample more frequently and/or be free to investigate otheraspects of the arthropods that they are studying. This scanning systemalso offers public agencies a quick and simple way of identifying commoninsects where people trained in taxonomy are not available or their timeis limited. If the individual is attempting to identify an uncommonspecies that is not in the classifier's database, the software can beinstructed to indicate that the best match is not a very good match(just like the method of clutter rejection) and that the user shouldconsider that the correct insect may be in another database or that ataxonomic expert should be consulted since this is likely to be anuncommon or poorly known species or even a previously unknown species.

Experiment 3.

This experiment further demonstrated the validity of general concepts ofsome embodiments of the invention and their application to a digitalcamera-based system. This configuration is applicable to the use of thetechnology of some embodiments in field detection stations, where theautomatic detection, identification and counting of insect/arthropodscaptured on or in various types of traps such as colored sticky boardsor baited pheromone traps is proposed. This configuration would include:

-   -   a) a sticky surface to which insects are attracted by various        stimuli including color, pheromones, kairomones or patterns; and    -   b) an imaging device to acquire images of the sticky surface at        various intervals. Processing of the images could be done in        situ or sent by various methods (cable, radio) to a processing        location.

Equipment setup for some embodiments. FIG. 39.

-   -   1. A digital video camera (Kodak MDS 100 Color Camera) with a        wide angle C-Mount lens (Computar 8.5 mm fixed focal length,        model M8513, with a 41.0 degree angular field of view for a ½        inch CCD) was mounted on a tripod. The lens was fitted with an        infrared filter to insure that the elements of the camera's        charged-coupled device (CCD) were exposed primarily to visible        light. The camera's lens was 26.04 cm (10.25 inches) from the        surface. With the digital zoom of the camera set to a        magnification of 1.5, the field of view was 7.9 cm by 5.9 cm        (3.1×2.3 inches). The lens has manual focus and iris rings. The        resolution of the Kodak MDS is 640×480 pixels.    -   2. A yellow surface (detection surface 624) (plastic back of        compact disk painted with fluorescent yellow paint ACE GLO Spray        Fluorescent).    -   3. Two incandescent lights (40 Watts). The height of the lamps        over the surface was 24.1 cm (9.5 inches).    -   4. A notebook computer (IBM Thinkpad 600) was used to store        image data from the camera. The camera was connected to the        computer via a Universal Serial Bus (USB). The computer was used        also to control the camera (shutter speed, digital zoom,        contrast, color balance, hue, saturation and brightness) and do        the processing for the detection and identification of the        arthropods.

Description of the Experiment and its Results

The same sequence of functions described previously in the section,GENERAL DESCRIPTION OF THE OPERATION OF THE SYSTEM, and used forExperiments 1, 2A and 2B, is also used here. For the purpose of claritythey are briefly repeated here as applied to this configuration.

A. Generation of a Background Image

An image of the yellow surface, (detection surface 624), was generatedprior to placing any insects to be identified on it. This image wassaved as a computer file, backg0.bmp (FIG. 40). To capture this imagethe computer mouse of the host computer was simply clicked on the iconfor Kodak's MDS100 software package. From the window of this program themouse was clicked on the “Take Picture” button and then from the Filemenu selected the command “Save As.”

B. Generation of Identifying Reference Features from Known Arthropods

Features were extracted from known insects, cotton boll weevils. Thesefeatures are utilized by the classifier of the software. The featureswere generated in a mode that is referred to as the “training mode” ofthe system. The detection and classification software, BugClassify.exe,was executed on an image of the known prototypes or reference bollweevils. The software is executed exactly in the same manner as the“detection/classification mode” until the last function, classification.The software then saves the feature set for each of the known insects toa file.

The equipment acquired an image of the reference weevils, calledTrain1.bmp (FIG. 41), and used this image along with the backgroundimage, backg0.bmp, to detect the reference weevils and to generate afeature file (Identifying Reference Feature file), called Weevil.txt.This file contains the values of each feature (feature set) extractedfrom each of the known boll weevils. This file was then edited toinclude the species and aspect/orientation that was associated with eachreference specimen's feature set. The picture, FIG. 41, contains sevencotton boll weevils used for training placed on a yellow surface invarious aspects (positions), which were as follows:

-   -   1) three on their sides;    -   2) one on its back;    -   3) one on its abdomen;    -   4) one partially on its side and back; and    -   5) one sitting on its posterior end.

When the software was run in the “training mode” silhouettes of eachreference weevil for classification were optionally not generated, sincesilhouette matching was not necessary for this application. Although allthe statistical features were calculated that were mentioned in SectionB of the earlier section, GENERAL DESCRIPTION ON OPERATING THE SYSTEM,an option was selected to write only the four most promising featuresfor identification to the file, Weevil.txt. The first two features weresize related, the third was a shape feature, and the fourthcharacterized the colors of the weevil:

-   -   Size-related features:        -   1) total area;        -   2) perimeter;    -   Shape-related feature:        -   3) circular fit or compactness feature—this feature was            described in the equivalent section of experiment 1.    -   Color feature:        -   4) the 2D hue/saturation color histogram of some            embodiments—this feature was also described in the            equivalent section of Experiment 1.

C. Acquisition of Images of the Unknown Arthropods to be Detected

To test the ability of the equipment, process and software to detectvarious unknown insects, two pictures were taken. These images simulatedthe actual use of the system to detect and identify insects on asurface. For the first of these pictures three weevils were placed onthe yellow surface (FIG. 42). This picture was saved as an electronicfile called, wst0.bmp. A second test image was taken (FIG. 43) thatincluded the previous three weevils plus two more weevils and acantharid beetle. This was stored as a file called, wst1.bmp.

D.-F. Arthropod Detection, Feature Extraction, Classification

This function involved labeling those pixels from the weevil images thatappeared different from the background image and thus were likely tobelong to a weevil or clutter. The labeled pixels were then connectedinto continuous regions or blobs by connected-components analysis.Regions that were too small in area were discarded. Features were thenextracted and compared with feature sets of known specimens via thesingle-nearest-neighbor classifier. Although the feature file containedthe values for the four previously described features, only two wereused to identify the cotton boll weevils: area and the 2D hue/saturationcolor histogram. For each unknown, the percentage difference in area andpercentage difference in overlap of the 2D histogram with respect toeach prototype in the feature file was calculated. The unknown wasassigned to the class of the prototype that was closest with respect toarea and distribution of colors. If, however, the best match differed bymore than 45% in area, or if the contents of the two 2D hue/saturationhistograms overlapped by less than 40%, this embodiment concluded thatthe match was not good and that the object must be something that hadnot been presented to the classifier during training (a species orobject not represented among the prototypes of the feature file). Inthis case, the unknown was assigned to a class called OTHER. Thesethresholds were empirically arrived at by prior testing with sets ofdifferent insects.

The first test image (FIG. 42) was analyzed by running the softwareBugClassify.exe with the images depicted in FIG. 40 and FIG. 42 and thefeature file, Weevil.txt, as input. FIG. 40 represents the previousbackground state while FIG. 42 is the image containing the three insectsto be detected and identified. FIG. 44 is an output from that process.All three boll weevils were detected with no false detects. The detectedor labeled pixels associated with each of the detected insects werereplaced in the output image with the color code for the species classthat was assigned by the classifier. If the classifier decided that anunknown arthropod was a cotton boll weevil, each of the pixels that wereassociated with that unknown by the segmentation process was coloredgreen in the output image. Pixels that were associated with an unknownthat was assigned to the class OTHER were colored red in the outputimage. Background pixels had the same values as the input image.

The software BugClassSilh.exe was executed for the second test image(FIG. 43), but this time FIG. 40 and FIG. 43 plus the feature file,Weevil.txt, were used as input. FIG. 45 is an output from that process.The weevils are identified as such according to their color code (green)and the cantharid is identified as the class OTHER (color coded red).FIG. 46 illustrates how close the unknown or test boll weevils are to areference boll weevil in feature space. This figure also shows howdifferent the cantharid is from the reference boll weevils in terms ofboth area and color. The cantharid beetle was rejected as a boll weevilbecause the area of the best weevil match differed from the cantharidbeetle by more than 45% and the color histograms overlapped by less than40%. However, if a statistical outlier test is used instead, such asGrubbs' test, a confidence level can be assigned to the best match. Inthis case, the best match for the cantharid beetle can be rejectedbecause Grubbs' test indicates that there is less than a 1% probabilitythat the cantharid is from the same population as the reference bollweevils based on area alone. Therefore, it can be concluded that thecantharid beetle does not belong to a class of any of the referencespecimens and should be labeled as OTHER. It would have been possible toidentify the cantharid as such if the system had been previously trainedto identify cantharids by including the feature values of one or morereference cantharids. However, in some embodiments the concern is onlywith counting the number of boll weevils.

These tests have again demonstrated the validity and practicality of theinvention's concepts. It was shown that the invention is able to detectinsects within an image. It was shown that it is possible to generatedistinguishing features to recognize insects and to recognize otherobjects for which the classifier was not trained. Objects that were notintended to be detected and counted were appropriately assigned to aclass called OTHER or unknown. It was also shown that by using theimage-processing algorithms of some embodiments, a digital color cameraand a computer system, it is possible to automate the detection andclassification of insects and other arthropods.

The various method embodiments of the present invention can beimplemented on a programmed computer, hardware circuit, or otherinformation-processing apparatus. As such, they are referred to as“machine-implemented methods.”

Some embodiments of the invention provide an apparatus that includes aninput device configured to receive image information, a detectorconfigures to distinguish one or more objects, including a first objectfrom a background of the image, a histogram generator that generateshistogram information for the first detected object, and a comparingdevice that compares the histogram information to each on of a pluralityof stored histogram records in order to generate an identification ofthe object.

In some embodiments, the object is an arthropod.

In some embodiments, the object includes a plurality of partiallyoverlapped arthropods to be distinguished from one another.

Some embodiments provide a machine-implemented method that includesacquiring a digital image; and detecting a first arthropod object in theimage, wherein the detecting includes distinguishing the first objectfrom a background image using image information selected from a groupconsisting of luminance, hue, color-saturation information andcombinations thereof.

In some embodiments, the image information used to distinguish the firstobject from the background includes luminance, hue and color-saturationinformation.

Some embodiments further include detecting a second object in the image,wherein the second object is at least partially overlapped with thefirst object, and distinguishing the first object from the second objectusing image information selected from a group consisting of luminance,hue, color-saturation information and combinations thereof.

In some embodiments, the second object is not an arthropod object.

Some embodiments further include detecting a second object in the image,and distinguishing a type of the first object from a type of the secondobject using image information selected from a group consisting ofluminance, hue, color-saturation information and combinations thereof.

In some embodiments, the type of the second object is not an arthropodtype.

Some embodiments further include generating first-object histograminformation based at least in part on color information of the detectedfirst object, and classifying a type of the first object based on thefirst object histogram information and storing a categorizationidentifier based on the classifying.

In some embodiments, the first-object histogram information is generatedbased on image information selected from a group consisting ofluminance, hue, color-saturation information and combinations thereof,and wherein the categorization identifier includes a genusidentification and a species identification.

In some embodiments, the acquiring of the image includes filtering lightfor the image to limit a spectral range of the light.

In some embodiments, the acquiring of the image includes filtering lightfor the image to limit a polarization of the light, and wherein theimage information used to distinguish the first object from thebackground includes luminance, hue and color-saturation information.

Some embodiments provide an information-processing apparatus thatincludes an input device coupled to receive a digital image, and adetector that detects a first arthropod object in the image, wherein thedetector includes a comparator operable to compare image informationselected from a group consisting of luminance, hue, color-saturationinformation and combinations thereof, and wherein the detectordistinguishes the first object from a background image based on thecomparison.

In some embodiments, the image information used by the comparatorincludes hue and color-saturation information.

In some embodiments, the detector further detects a second object in theimage, wherein the second object is at least partially overlapped withthe first object, and the detector distinguishes the first object fromthe second object based on a comparison of image information selectedfrom a group consisting of luminance, hue, color-saturation informationand combinations thereof.

In some embodiments, the second object is not an arthropod object.

In some embodiments, the detector also detects a second object in theimage, and distinguishes a type of the first object from a type of thesecond object using image information selected from a group consistingof luminance, hue, color-saturation information and combinationsthereof.

In some embodiments, the type of the second object is not an arthropodtype.

Some embodiments further include an identifier that associatescategorization identification with the first object.

In some embodiments, the categorization identification includes a genusidentification and a species identification.

Some embodiments further include an image-acquisition device thatincludes a filter to limit a spectral range of acquired light.

Some embodiments further include an image-acquisition device thatincludes a filter to limit a polarization of acquired light, and whereinthe image information used to distinguish the first object from thebackground includes luminance, hue and color-saturation information.

Some embodiments provide a classifier that can recognize the arthropodsregardless of how they are oriented with respect to the imaging device,and in addition to classifying arthropods the system can recognizenon-arthropod objects or clutter, image artifacts such as shadows andglare, occlusion or overlapping and touching objects, and incompletearthropods and the system optionally including one or more of thefollowing:

-   -   a) an imaging device to capture pictures of arthropods and the        device may be chosen from among the following image sensor        types: digital camera, digital scanner, analog or digital video        camera; and the sensor should collect color imagery, but a black        and white sensor can be substituted for the purpose of reducing        cost.    -   b) an appropriate camera lens for optically coupled to the image        device to insure sufficient magnification of the insects and a        practical field of view.    -   c) one or more lens filters to select the portion of the light        spectrum that is most efficient for detecting the arthropods of        concern and/or filter(s) to selectively remove non-polarized        light to reduce glare.    -   d) a box-like lid for a scanner to prevent contact of the        scanner's lid with the arthropods.    -   e) a polarizing filter placed on the scanner surface to reduce        glare by removing non-polarized light.    -   f) illumination device such as a LED illuminator, ring light or        high intensity flash to insure uniform and similar lighting        conditions for each image captured and where possible to reduce        shadows and glare.    -   g) a communication link between the camera and the processor,        which can include: a direct cable connection using a Universal        Serial Bus (USB) connection or a RS-232 serial port connection;        wireless device using a radio or infra-red communications band;        a phone modem; or an internet connection.    -   h) a processor along with sufficient memory, and operating        system and software to control the camera's functions including        lighting and color settings, requesting the capture and transfer        of images, processing the image(s) for the detection and        identification of arthropods and printing out and/or displaying        the results; said processor can be a general purpose computer or        specialized computing hardware designed for the arthropod        detection and identification system.    -   i) software to adjust camera settings, capture an image, adjust        parameters for image processing routine, apply image processing        techniques for the detection and identification of the        arthropods, display results to a computer monitor, save results        to a computer file, and/or edit files.    -   j) a surface to place or capture the arthropods that allows the        imaging device a clear view to collect images and this surface        can include a simple stand alone inspection surface or a surface        that is part of a trap or collection device.

Some embodiments provide a first method implemented in software thatautomatically detects objects including arthropods in an image usingluminance, hue and color saturation information to distinguish theobjects from a background or an estimated background image.

Some embodiments provide a second method implemented in software thatautomatically rejects shadows by examining differences in luminance, hueand color saturation between the background or estimated backgroundimage and the image being checked for arthropods.

Some embodiments provide a third method implemented in software toextract statistical features that characterize an object's size, shape,luminance and colors (which in the case of reference specimens ofarthropods can be stored to a computer file or database) and can be usedto calculate a mean and standard deviation for each feature from amongthe reference specimens of a species, which is to be similarly storedwith the features.

Some embodiments provide a fourth method implemented in software toextract: 1) an object's silhouette or outer profile; 2) distinguishinginternal edges due to large gradients in luminance or color; 3)reference points of a known offset from the silhouette containing hueand saturation information; and 4) the prototype or referencesilhouettes and color samples of arthropods can be stored to a computerfile or database.

Some embodiments provide a fifth method or statistical classifierimplemented in software that automatically compares statistical featuresextracted by the third method just described from reference specimens ofarthropods and the features similarly extracted from the unknown objectunder consideration.

Some embodiments provide a sixth method that, on the basis of theunknown's set of features and those of the reference specimens, findsthe class which the unknown object is mostly likely to be a member of.

Some embodiments provide a seventh method to assign a statisticalconfidence to classifier's decision by comparing how each of thefeatures of the unknown are distributed relative to the mean andstandard deviation of the features belonging to the members of the classof the matched reference specimen.

Some embodiments provide an eighth method to use the confidence level tomake a final decision which can be either: 1) accept the class of thebest match for the unknown if the confidence level is good; 2) rejectthe best match and assign the unknown to an undeterminable class notrepresented by the reference specimens when one or more features of theunknown exceed the confidence level associated with the class of thebest match; 3) as an alternative to item 2, rather than reject theunknown as undeterminable when there is a low confidence instead passthe decision making on to a higher-level syntactic or structural patternrecognition that can deal with occlusion, missing arthropod features,and clutter.

Some embodiments provide a ninth method or classification processimplemented to run automatically in software that compares the prototypesilhouettes and associated line edges of arthropod structures andcolor-sample point-of-reference specimens extracted by the fourth method(just described above) with the silhouette and associated information ofthe unknown.

Some embodiments provide a tenth method and/or logic to iterativelytranslate, rotate and scale each prototype silhouette looking for thebest match and record other good matches for the silhouette and repeatthis process for each prototype silhouette.

Some embodiments provide an eleventh method and/or logic that assignsthe best silhouette/color sample match to the detection or portion ofthe detection, provided that the silhouettes and color samples fit theunknown well or otherwise assigns the detected area as clutter.

Some embodiments provide a twelfth method and/or logic that repeats theprocess for other portions of the detection that have not been explainedby any previous silhouette matching until all the detection's area hasbeen explained as being part of an arthropod(s) or clutter.

Some embodiments provide a thirteenth method and/or logic that takes thefinal results from the process (or subcombinations of the process)defined above and updates the species count of the arthropods andclutter that have been detected and identified by the system.

Some embodiments provide a fourteenth method implemented in software toautomatically report the detected and identified arthropods and clutter,and to provide a summary of the detections and identifications to auser's screen, computer file, and/or to output a graphic representationto an image file that is saved to memory or displayed to the user'sscreen.

Some embodiments provide a fifteenth method implemented in software toallow the user to interact with the software.

Some embodiments provide a method to alter various parameters of thedetection and classification process to allow the user to adapt theprocess to special situations.

Some embodiments provide a sixteenth method to allow the user to requestthe saving or output of various intermediate results, such as a detectedpixels image, segmentation or labeled image of detections, silhouetteimage.

Some embodiments provide a seventeenth method to alter the settings ofthe image device. For example, automatic analysis of the image canprovide feedback to the image device to improve subsequent images.Alternatively, the human user can look at the image, and provide inputto adjust the settings of the imaging device or the illuminationprovided.

Some embodiments provide a eighteenth method to request the capture ofan image or the scheduling of periodic captures of images along with theprocessing of those images for the detection and identification ofarthropods.

Some embodiments provide a nineteenth method to support the system withoff-line editing of the reference feature and silhouette files toassociate the appropriate class identity with each feature set orsilhouette.

Some embodiments provide combinations of two or more of the firstthrough nineteenth methods just described, or of subportions of thesemethods. These combinations do not necessarily require that any one ofthe methods be either included or omitted. Some combinations furtherinclude other processes, methods, or portions thereof describedelsewhere herein. Some embodiments of the invention include (see FIG. 3)a computer-readable media 321 (such as a diskette, a CDROM, a DVDROM,and/or a download connection to the internet) having instructions storedthereon for causing a suitably programmed data processor to execute oneor more of the above methods.

Some embodiments provide various supplies that enhance thearthropod-capture process, and/or the image-acquisition process. Forexample, some embodiments provide an arthropod-capture substrate thatincludes a sticky surface, and is also colored. In some embodiments, thesubstrate is colored to attract arthropods of interest. In someembodiments, two or more contrasting colors are provided in order toprovide better contrast for a first type of arthropod on a first color,and better contrast for a second type of arthropod on a second color. Insome embodiments, a plurality of different colors and/or gray scalesand/or hues and saturations are provided (either as part of thesubstrate, or as an ancillary surface that will be imaged with thesubstrate and the arthropods), in order to provide calibrationinformation (color (such as hue and saturation), brightness, and/orcontrast) in each captured image. In some embodiments, the substrateincludes a chemical attractant. In some embodiments, the chemicalattractant is supplied as a separate source (e.g., a carbon dioxidecontainer such as a gas cylinder, or supplied from a generator or flame,in order to attract mosquitos or other anthropods) wherein the chemicalis emitted through or near the sticky capture surface of the substrate.

It is understood that the above description is intended to beillustrative, and not restrictive. Many other embodiments will beapparent to those of skill in the art upon reviewing the abovedescription. The scope of the invention should, therefore, be determinedwith reference to the appended claims, along with the full scope ofequivalents to which such claims are entitled. In the appended claims,the terms “including” and “in which” are used as the plain-Englishequivalents of the respective terms “comprising” and “wherein,”respectively. Moreover, the terms “first,” “second,” and “third,” etc.,are used merely as labels, and are not intended to impose numericalrequirements on their objects.

1. An apparatus comprising: an input device coupled to receive a digitalimage; and a detector, coupled to the input device, that detects anobject in the image; a histogram generator coupled to receive objectimage information from the detector, that generates a first object-imagehistogram based at least in part on image information of the detectedobject, wherein the first object-image histogram has a plurality ofbins, each one of the plurality of bins located on: a row correspondingto a value of a first color-space dimension, and a column correspondingto a value of a second color-space dimension; and a classifier thatclassifies the object based on the first image-object histogram andprovides a categorization identifier based on the classification.
 2. Theapparatus of claim 1, wherein the histogram generator uses colorinformation to generate the first image-object histogram, including hueinformation for the first dimension and color-saturation information forthe second dimension.
 3. The apparatus of claim 1, wherein histogramgenerator generates the first image-object histogram having the firstdimension that includes hue and the second dimension that includescolor-saturation.
 4. The apparatus of claim 1, wherein the detector usescolor information to detect the object, including both hue informationand color-saturation information.
 5. The apparatus of claim 1, whereinthe histogram generator uses color information based on calculations,from the red, green, blue (RGB) values of each of a plurality of pixels,of substantially: CR=0.701×RED+0.587×GREEN+0.114×BLUE,CB=−0.299×RED+0.587×GREEN+0.886×BLUE, SATURATION=SQUARE ROOT(CRSQUARED+CB SQUARED), and HUE=ARCTAN(CR/CB).
 6. An apparatus comprising:an input device coupled to receive a digital image; and a detector,coupled to the input device, that detects an object in the image; ahistogram generator, coupled to receive object image information fromthe detector, that generates object-image-histogram information based atleast in part on image information of the detected object; a classifierthat classifies the object based on the object-image-histograminformation and provides a categorization identifier based on theclassification; and a distinguisher that distinguishes a first arthropodin the detected object from a second item in the detected object, indigital images where the detected object includes the first arthropod atleast partially overlapped with the second item, using image informationselected from a group consisting of luminance, hue, color-saturationinformation and combinations thereof.
 7. An apparatus comprising: aninput device coupled to receive a digital image; and a detector, coupledto the input device, that detects an object in the image; a histogramgenerator, coupled to receive object image information from thedetector, that generates object-image-histogram information based atleast in part on image information of the detected object; a classifierthat classifies the object based on the object-image-histograminformation and provides a categorization identifier based on theclassification; and a imagable surface having at least one sticky areafor capturing arthropods and having at least one colored area forcalibrating colors of the image.
 8. A machine-implemented methodcomprising: acquiring a digital image; detecting a first arthropodobject in the image; and generating first-object histogram informationbased at least in part on color information of the detected firstobject; and classifying a type of the first object based on thefirst-object histogram information and generating a categorizationidentifier based on the classifying.
 9. The method of claim 8, whereinthe color information used to generate the histogram includes hue andcolor-saturation information.
 10. The method of claim 8, furthercomprising: detecting a second object in the image, wherein the secondobject is at least partially overlapped with the first object;distinguishing the first object from the second object using imageinformation selected from a group consisting of luminance, hue,color-saturation information and combinations thereof.
 11. The method ofclaim 8, wherein the first-object histogram information is generatedbased on hue and color-saturation information, and wherein thecategorization identifier includes a genus identification and a speciesidentification.
 12. The method of claim 8, wherein the acquiring of theimage includes filtering light for the image to limit a spectral rangeof the light.
 13. The method of claim 8, wherein the acquiring of theimage includes filtering light for the image to limit a polarization ofthe light.
 14. The method of claim 8, wherein the acquiring of the imageincludes providing a sticky substrate having first background area thathas a first color that is darker than the first object and a secondbackground area that has a second color that is lighter than the firstobject.
 15. The method of claim 8, wherein the acquiring of the imageincludes providing a substrate having first area that has a firstcalibration color, and a second area that has a second calibration colorthat is different from the first calibration color.
 16. The method ofclaim 8, further comprising: distinguishing the first object from abackground image using image information selected from a groupconsisting of hue information, color-saturation information andcombinations thereof.
 17. The method of claim 8, further comprising:adding the categorization identifier to a database, the databaseincluding information on a date and a location associated with thecategorization identifier.
 18. The method of claim 8, furthercomprising: applying a pest-control protocol based on one or morecategorization identifiers of the database.
 19. An apparatus comprising:an input device coupled to receive a digital image; means for detectinga first arthropod object in the image and distinguishing the firstobject from a background image using image information selected from agroup consisting of luminance, hue, color-saturation information andcombinations thereof; and means for classifying the detected firstarthropod object that includes generating a histogram having a pluralityof independent dimensions including at least a first dimension and asecond dimension.
 20. The apparatus of claim 19, farther comprising:means for detecting a second object in the image, wherein the secondobject is at least partially overlapped with the first object, and fordistinguishing the first object from the second object using imageinformation selected from a group consisting of luminance, hue,color-saturation information and combinations thereof.
 21. The apparatusof claim 20, wherein the second object is not an arthropod object. 22.The apparatus of claim 19, further comprising means for acquiring theimage including a filter to limit a spectral range of the light and tolimit a polarization of the light, and wherein the image informationused to distinguish the first object from the background includesluminance, hue and color-saturation information.